Jul 09 18:01:42.973052 ip-10-0-133-76 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 18:01:42.973062 ip-10-0-133-76 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 18:01:42.973069 ip-10-0-133-76 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 18:01:42.973310 ip-10-0-133-76 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 18:01:53.165670 ip-10-0-133-76 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 18:01:53.165685 ip-10-0-133-76 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5806bec92a4347348d470d788b0e3c96 --
Jul 09 18:03:47.457696 ip-10-0-133-76 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:03:47.760913 ip-10-0-133-76 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:47.760913 ip-10-0-133-76 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:03:47.760913 ip-10-0-133-76 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:47.760913 ip-10-0-133-76 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:03:47.760913 ip-10-0-133-76 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:47.761683 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.761598    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:03:47.765631 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765614    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:47.765631 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765630    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765634    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765639    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765642    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765645    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765648    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765651    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765654    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765657    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765660    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765662    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765665    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765667    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765670    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765672    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765676    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765688    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765691    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765693    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765696    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:47.765697 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765699    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765702    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765705    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765708    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765711    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765714    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765716    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765719    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765722    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765725    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765730    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765733    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765736    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765739    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765742    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765745    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765747    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765750    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765752    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:47.766193 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765755    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765759    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765762    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765765    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765768    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765770    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765773    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765776    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765778    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765781    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765784    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765786    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765789    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765792    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765794    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765798    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765800    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765803    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765805    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:47.766658 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765808    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765810    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765813    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765815    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765818    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765820    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765823    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765826    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765828    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765831    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765833    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765836    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765838    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765841    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765843    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765845    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765848    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765851    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765853    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765855    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:47.767136 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765858    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765861    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765863    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765866    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765868    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765872    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765876    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765879    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765883    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765885    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765888    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765891    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765893    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765896    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765898    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765901    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765903    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765906    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765908    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.765911    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:47.767613 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766645    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766653    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766657    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766660    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766664    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766667    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766669    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766672    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766675    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766678    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766681    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766684    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766686    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766689    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766693    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766695    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766698    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766701    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766705    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:47.768103 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766707    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766711    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766713    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766716    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766718    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766721    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766724    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766726    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766729    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766732    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766735    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766737    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766740    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766743    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766745    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766748    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766750    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766753    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766755    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766758    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:47.768571 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766760    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766762    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766765    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766768    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766770    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766773    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766775    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766778    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766780    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766783    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766785    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766788    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766793    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766797    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766800    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766805    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766808    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766810    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766813    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766816    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:47.769085 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766818    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766821    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766823    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766826    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766829    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766831    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766834    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766837    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766839    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766842    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766844    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766847    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766849    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766852    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766854    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766856    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766859    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766861    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766864    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766866    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:47.769579 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766869    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766872    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766874    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766877    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766879    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766882    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766885    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766890    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766894    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766897    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766901    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766904    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766907    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766910    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766912    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766915    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766917    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766920    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766922    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:47.770058 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.766925    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767304    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767310    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767313    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767315    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767318    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767321    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767324    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767327    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767329    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767332    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767334    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767337    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767339    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767342    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767345    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767348    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767351    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767354    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:47.770534 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767357    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767359    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767362    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767365    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767367    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767370    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767373    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767375    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767378    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767382    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767386    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767389    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767392    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767394    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767397    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767400    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767415    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767417    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767421    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:47.770989 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767424    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767427    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767429    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767432    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767435    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767437    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767440    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767443    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767445    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767448    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767451    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767454    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767457    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767460    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767463    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767466    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767468    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767471    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767474    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767477    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:47.771471 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767480    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767483    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767485    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767488    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767491    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767493    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767496    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767498    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767501    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767503    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767506    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767508    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767511    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767513    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767516    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767519    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767521    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767523    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767526    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767529    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:47.771944 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767532    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767534    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767540    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767543    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767547    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767551    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767553    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767557    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767561    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767564    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767567    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767569    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767572    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767575    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767577    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767580    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767582    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767585    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767587    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:47.772419 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767590    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767593    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.767595    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767666    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767673    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767679    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767684    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767688    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767691    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767695    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767699    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767702    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767705    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767709    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767712    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767717    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767720    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767725    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767728    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767731    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767734    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767738    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767741    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767744    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:03:47.772870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767747    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767750    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767754    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767757    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767760    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767764    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767767    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767770    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767773    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767776    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767779    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767783    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767786    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767789    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767791    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767794    2563 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767798    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767802    2563 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767805    2563 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767808    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767811    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767814    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767817    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767822    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767825    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:03:47.773460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767830    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767833    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767836    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767839    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767842    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767845    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767847    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767850    2563 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767854    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767857    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767860    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767863    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767866    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767869    2563 flags.go:64] FLAG: --help="false"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767872    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-76.ec2.internal"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767875    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767877    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767880    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767883    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767887    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767889    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767892    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767895    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:03:47.774056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767898    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767901    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767904    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767907    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767910    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767912    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767915    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767919    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767922    2563 flags.go:64] FLAG: --lock-file=""
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767926    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767931    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767934    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767939    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767942    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767945    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767948    2563 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767950    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767954    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767957    2563 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767959    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767963    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767966    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767970    2563 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767973    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767976    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:03:47.774712 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767979    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767982    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767985    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767988    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767991    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.767998    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768001    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768004    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768007    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768010    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768015    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768018    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768022    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768024    2563 flags.go:64] FLAG: --port="10250"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768030    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768032    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-001707513e7d99489"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768036    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768041    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768044    2563 flags.go:64] FLAG: --register-node="true"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768047    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768051    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768054    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768057    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768060    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768064    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:03:47.775307 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768067    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768070    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768073    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768076    2563 flags.go:64] FLAG: --runonce="false"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768078    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768081    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768084    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768101    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768105    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768108    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768112    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768115    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768118    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768121    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768124    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768127    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768130    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768133    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768136    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768141    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768144    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768148    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768153    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768156    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768160    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:03:47.775929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768163    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768166    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768169    2563 flags.go:64] FLAG: --v="2"
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768176    2563 flags.go:64] FLAG: --version="false"
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768180    2563 flags.go:64] FLAG: --vmodule=""
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768183    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.768186    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768305    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768308    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768312    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768315    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768317    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768320    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768324    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768327    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768330    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768333    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768336    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768339    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768342    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768345    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:47.776542 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768347    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768350    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768352    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768355    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768357    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768360    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768363    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768367    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768369    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768372    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768376    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768378    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768381    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768384    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768387    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768390    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768392    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768395    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768397    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768400    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:47.777028 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768415    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768418    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768420    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768423    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768425    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768428    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768431    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768434    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768438    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768442    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768445    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768448    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768451    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768454    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768457    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768459    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768462    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768465    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768468    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:47.777522 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768472    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768475    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768477    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768482    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768485    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768487    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768490    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768494    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768498    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768501    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768504    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768507    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768510    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768513    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768516    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768518    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768521    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768525    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768527    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:47.778012 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768531    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768534    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768537    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768540    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768543    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768545    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768548    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768550    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768553    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768556    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768558    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768561    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768563    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768567    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768570    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768573    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768578    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768580    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768583    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768586    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:47.778481 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768589    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768592    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768595    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768597    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768600    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768603    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.768605    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.769316    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.776183    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.776200    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.776219    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.776228    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776288    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776293    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776298    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:47.778973 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776301    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776304    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776307    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776309    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776313    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776315    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776318    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776321    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776323    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776326    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776328    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776331    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776333    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776336    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776339    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776341    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776344    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776347    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776349    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776352    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:47.779331 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776354    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776357    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776360    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776362    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776365    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776367    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776377    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776379    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776382    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776385    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776388    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776390    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776393    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776395    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776398    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776415    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776418    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776422    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776426    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776429    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:47.779965 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776433    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776436    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776438    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776442    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776445    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776447    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776450    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776453    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776457    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776460    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776463    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776466    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776470    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776472    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776475    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776478    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776481    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776483    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776487    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:47.780451 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776496    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776500    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776502    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776505    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776507    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776510    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776512    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776515    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776517    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776519    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776522    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776524    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776527    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776530    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776532    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776534    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776537    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776539    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776542    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:47.780908 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776546    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776549    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776552    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776554    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776558    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776561    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776564    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776566    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776569    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776571    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776574    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776576    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776579    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776582    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776590    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776593    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776595    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:47.781353 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776598    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.776603    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776748    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776753    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776756    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776758    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776761    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776764    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776767    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776770    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776772    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776775    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776777    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776780    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:47.781782 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776783    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776786    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776790    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776793    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776796    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776799    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776802    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776805    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776807    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776810    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776813    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776815    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776818    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776820    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776823    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776826    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776834    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776837    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776840    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:47.782125 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776842    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776845    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776848    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776850    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776853    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776855    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776858    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776860    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776863    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776865    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776867    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776871    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776875    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776878    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776880    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776883    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776886    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776888    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776891    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776894    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:47.782608 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776896    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776899    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776901    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776904    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776906    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776909    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776911    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776914    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776917    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776919    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776929    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776932    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776934    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776937    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776939    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776942    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776945    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776949    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776951    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776954    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:47.783072 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776956    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776959    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776961    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776964    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776966    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776968    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776971    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776974    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776976    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776979    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776981    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776984    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776986    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776989    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776991    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776994    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776996    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.776999    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777001    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777004    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:47.783569 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777006    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777009    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777011    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777019    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777022    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777025    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777027    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:47.777030    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.777035    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.777165    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.779317    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.780502    2563 server.go:1019] "Starting client certificate rotation"
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.780619    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.780659    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:03:47.784043 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.782811    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:03:47.788184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.788164    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:03:47.795629 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.795611    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:03:47.797635 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.797613    2563 log.go:25] "Validated CRI v1 image API"
Jul 09 18:03:47.798519 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.798492    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:03:47.800716 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.800697    2563 fs.go:135] Filesystem UUIDs: map[7829d144-d191-4f63-b76b-a43af7a16555:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 875337e6-dd1d-4393-b2fc-4ebbf65f6ea2:/dev/nvme0n1p4]
Jul 09 18:03:47.800767 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.800716    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:03:47.805536 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.805421    2563 manager.go:216] Machine: {Timestamp:2026-07-09 18:03:47.804267696 +0000 UTC m=+0.253850110 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100171 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e3dcd8d3c2a90ab0d83f79e03bbc5 SystemUUID:ec2e3dcd-8d3c-2a90-ab0d-83f79e03bbc5 BootID:5806bec9-2a43-4734-8d47-0d788b0e3c96 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ba:2c:d7:90:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ba:2c:d7:90:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:0b:df:93:4a:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:03:47.805536 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.805534    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:03:47.805636 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.805609    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:03:47.808106 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.808082    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:03:47.808242 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.808108    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-76.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:03:47.808755 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.808746    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:03:47.808795 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.808758    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:03:47.808795 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.808765    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:03:47.809306 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.809295    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:03:47.810060 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.810046    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:03:47.810174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.810162    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:03:47.814011 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.813999    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:03:47.814126 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.814117    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:03:47.814165 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.814145    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:03:47.814165 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.814156    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:03:47.814165 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.814166    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:03:47.814289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.814175    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:03:47.814289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.814183    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:03:47.814641 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.814628    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:03:47.814680 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.814653    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:03:47.817607 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.817584    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:03:47.819351 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.819337    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:03:47.819427 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.819367    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:03:47.819983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.819969    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:03:47.820039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.819990    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:03:47.820039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.819999    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:03:47.820039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820007    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:03:47.820039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820018    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:03:47.820039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820023    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:03:47.820039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820031    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:03:47.820039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820036    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:03:47.820039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820042    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:03:47.820238 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820048    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:03:47.820238 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820057    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:03:47.820238 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820063    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:03:47.820627 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820615    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:03:47.820627 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.820625    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:03:47.823983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.823960    2563 server.go:1268] "Started kubelet"
Jul 09 18:03:47.826580 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.824999    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:03:47.826580 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.825823    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:03:47.826580 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.825931    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:03:47.825050 ip-10-0-133-76 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:03:47.827264 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.827189    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:03:47.827352 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.827256    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-76.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:03:47.827774 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.827760    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:03:47.828134 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.828087    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:03:47.832594 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.832574    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:03:47.833005 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.832984    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:03:47.833529 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.832816    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-76.ec2.internal.18c0b125ada49727  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-76.ec2.internal,UID:ip-10-0-133-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-76.ec2.internal,},FirstTimestamp:2026-07-09 18:03:47.823867687 +0000 UTC m=+0.273450104,LastTimestamp:2026-07-09 18:03:47.823867687 +0000 UTC m=+0.273450104,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-76.ec2.internal,}"
Jul 09 18:03:47.833788 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.833771    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:03:47.833889 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.833789    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:03:47.834002 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.833982    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:47.834367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834353    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:03:47.834453 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834358    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:03:47.834453 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834395    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:03:47.834545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834479    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:03:47.834545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834487    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:03:47.834720 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834706    2563 factory.go:55] Registering systemd factory
Jul 09 18:03:47.834793 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834723    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:03:47.834955 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834937    2563 factory.go:153] Registering CRI-O factory
Jul 09 18:03:47.835013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.834958    2563 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:03:47.835066 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.835011    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:03:47.835066 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.835040    2563 factory.go:103] Registering Raw factory
Jul 09 18:03:47.835066 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.835054    2563 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:03:47.835565 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.835550    2563 manager.go:349] Starting recovery of all containers
Jul 09 18:03:47.838452 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.838427    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zrtx6"
Jul 09 18:03:47.839631 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.839451    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:03:47.839631 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.839522    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-76.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:03:47.845503 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.845485    2563 manager.go:354] Recovery completed
Jul 09 18:03:47.846988 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.846888    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zrtx6"
Jul 09 18:03:47.847307 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.847289    2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 09 18:03:47.850296 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.850282    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:47.852177 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.852156    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:47.852241 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.852189    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:47.852241 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.852200    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:47.852794 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.852780    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:03:47.852794 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.852788    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:47.852794 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.852792    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:03:47.852932 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.852807    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:03:47.853447 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.853418    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:47.853508 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.853455    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:47.853508 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.853466    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:47.854453 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.854375    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-76.ec2.internal.18c0b125af5485ea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-76.ec2.internal,UID:ip-10-0-133-76.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-76.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-76.ec2.internal,},FirstTimestamp:2026-07-09 18:03:47.852174826 +0000 UTC m=+0.301757239,LastTimestamp:2026-07-09 18:03:47.852174826 +0000 UTC m=+0.301757239,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-76.ec2.internal,}"
Jul 09 18:03:47.854827 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.854817    2563 policy_none.go:49] "None policy: Start"
Jul 09 18:03:47.854859 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.854832    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:03:47.854859 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.854843    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:03:47.856836 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.856825    2563 policy_none.go:47] "Start"
Jul 09 18:03:47.898980 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.898960    2563 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:03:47.899074 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.898992    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:03:47.899074 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.899001    2563 server.go:87] "Starting device plugin registration server"
Jul 09 18:03:47.899231 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.899220    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:03:47.899292 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.899235    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:03:47.899355 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.899325    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:03:47.899437 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.899425    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:03:47.899437 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.899437    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:03:47.899919 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.899901    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:03:47.900006 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.899944    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:47.965273 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.965236    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:03:47.966372 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.966349    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:03:47.966498 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.966391    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:03:47.966498 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.966434    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:03:47.966498 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:47.966470    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:03:47.968875 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.968856    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:47.999752 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:47.999721    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:48.000784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.000769    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:48.000860 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.000796    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:48.000860 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.000807    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:48.000860 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.000831    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.006460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.006445    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.006508 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.006467    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-76.ec2.internal\": node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.018381 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.018327    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.066534 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.066505    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"]
Jul 09 18:03:48.066643 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.066570    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:48.068823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.068808    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:48.068916 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.068837    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:48.068916 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.068847    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:48.071123 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.071111    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:48.071219 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.071204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.071254 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.071232    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:48.072013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.071999    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:48.072085 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.072029    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:48.072085 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.072043    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:48.072085 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.072061    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:48.072085 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.072083    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:48.072260 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.072097    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:48.074396 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.074378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.074486 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.074428    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:48.075106 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.075091    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:48.075169 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.075112    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:48.075169 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.075120    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:48.097014 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.096991    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-76.ec2.internal\" not found" node="ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.101276 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.101262    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-76.ec2.internal\" not found" node="ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.119357 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.119335    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.135558 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.135541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9722980453de5610e5dce33afc71c5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-76.ec2.internal\" (UID: \"9722980453de5610e5dce33afc71c5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.135634 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.135564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6779a578b4eee4a48a9308987768e0e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal\" (UID: \"6779a578b4eee4a48a9308987768e0e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.135634 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.135584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6779a578b4eee4a48a9308987768e0e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal\" (UID: \"6779a578b4eee4a48a9308987768e0e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.219543 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.219510    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.236219 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.236187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9722980453de5610e5dce33afc71c5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-76.ec2.internal\" (UID: \"9722980453de5610e5dce33afc71c5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.236303 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.236226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6779a578b4eee4a48a9308987768e0e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal\" (UID: \"6779a578b4eee4a48a9308987768e0e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.236303 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.236242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6779a578b4eee4a48a9308987768e0e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal\" (UID: \"6779a578b4eee4a48a9308987768e0e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.236303 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.236277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6779a578b4eee4a48a9308987768e0e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal\" (UID: \"6779a578b4eee4a48a9308987768e0e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.236303 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.236282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9722980453de5610e5dce33afc71c5c6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-76.ec2.internal\" (UID: \"9722980453de5610e5dce33afc71c5c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.236469 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.236300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6779a578b4eee4a48a9308987768e0e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal\" (UID: \"6779a578b4eee4a48a9308987768e0e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.320493 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.320438    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.398781 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.398745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.403719 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.403584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.421197 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.421178    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.521512 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.521480    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.621756 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.621699    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.682733 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.682710    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:48.721782 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.721753    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.780045 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.780021    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:03:48.780570 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.780157    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:03:48.780570 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.780174    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:03:48.822305 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:48.822276    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-76.ec2.internal\" not found"
Jul 09 18:03:48.841466 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.841446    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:48.851693 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.851667    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:58:47 +0000 UTC" deadline="2028-02-28 20:41:37.554009171 +0000 UTC"
Jul 09 18:03:48.851693 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.851690    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14378h37m48.702322255s"
Jul 09 18:03:48.934281 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.934256    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.944606 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.944585    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:03:48.945346 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.945332    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal"
Jul 09 18:03:48.954018 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.953999    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:03:48.984500 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:48.984468    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9722980453de5610e5dce33afc71c5c6.slice/crio-1f30aa31454e4875bc36dbed55ef5e237eb5214c6916a7ed819153034dd5d196 WatchSource:0}: Error finding container 1f30aa31454e4875bc36dbed55ef5e237eb5214c6916a7ed819153034dd5d196: Status 404 returned error can't find the container with id 1f30aa31454e4875bc36dbed55ef5e237eb5214c6916a7ed819153034dd5d196
Jul 09 18:03:48.986717 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:48.986694    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6779a578b4eee4a48a9308987768e0e1.slice/crio-f912cf94c782f89bb778c5f43fa2db123343fdd5c9365752f18dd553ee193a8f WatchSource:0}: Error finding container f912cf94c782f89bb778c5f43fa2db123343fdd5c9365752f18dd553ee193a8f: Status 404 returned error can't find the container with id f912cf94c782f89bb778c5f43fa2db123343fdd5c9365752f18dd553ee193a8f
Jul 09 18:03:48.989711 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:48.989690    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:03:49.351528 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.351440    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:49.802146 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.802115    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:49.816229 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.816207    2563 apiserver.go:52] "Watching apiserver"
Jul 09 18:03:49.824387 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.824363    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:03:49.825367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.825338    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gvpv7","openshift-multus/network-metrics-daemon-p55kt","openshift-network-operator/iptables-alerter-cpmn4","kube-system/global-pull-secret-syncer-j6fjk","kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal","openshift-network-diagnostics/network-check-target-ns4p4","openshift-ovn-kubernetes/ovnkube-node-p5dg8","kube-system/konnectivity-agent-9rrpv","openshift-cluster-node-tuning-operator/tuned-hzhjh","openshift-dns/node-resolver-rdjmn","openshift-image-registry/node-ca-msphn","openshift-multus/multus-additional-cni-plugins-zl75c"]
Jul 09 18:03:49.827976 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.827867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:03:49.830635 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.830584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.830846 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.830807    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l545p\""
Jul 09 18:03:49.831017 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.830996    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:03:49.831093 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.831018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:03:49.832875 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.832722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.832875 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:49.832793    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:03:49.834309 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.834283    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:03:49.835266 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.834895    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.835266 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.834929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.835266 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.835228    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.835266 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.835230    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:03:49.835586 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.835288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tsfs8\""
Jul 09 18:03:49.837602 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.837296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:49.837682 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.837617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.838082 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.838018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:03:49.838082 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.838068    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.838254 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.838023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gshv2\""
Jul 09 18:03:49.838254 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.838020    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:03:49.839377 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.839358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:49.839482 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:49.839438    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:03:49.839900 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.839881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.840099 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.840077    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:03:49.840197 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.840107    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7cn7m\""
Jul 09 18:03:49.840327 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.840313    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.842536 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.841726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.845160 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:03:49.845251 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845194    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:03:49.845372 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845351    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.845696 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845645    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w7ns2\""
Jul 09 18:03:49.845696 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-k8s-cni-cncf-io\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.845830 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-etc-kubernetes\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.845830 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c353489-1b68-489f-ace9-2e463f42d74b-host\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.845830 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.845983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-system-cni-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.845983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-hostroot\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.845983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-daemon-config\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.845983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dde90adf-88d2-4060-8384-205e8325b6b7-agent-certs\") pod \"konnectivity-agent-9rrpv\" (UID: \"dde90adf-88d2-4060-8384-205e8325b6b7\") " pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:03:49.845983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dde90adf-88d2-4060-8384-205e8325b6b7-konnectivity-ca\") pod \"konnectivity-agent-9rrpv\" (UID: \"dde90adf-88d2-4060-8384-205e8325b6b7\") " pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:03:49.846213 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.845970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c353489-1b68-489f-ace9-2e463f42d74b-serviceca\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2g5\" (UniqueName: \"kubernetes.io/projected/5c353489-1b68-489f-ace9-2e463f42d74b-kube-api-access-bq2g5\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-cni-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-os-release\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-cni-bin\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-cni-multus\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50672b24-2d0d-4afc-b7de-ac97da06cb99-kubelet-config\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50672b24-2d0d-4afc-b7de-ac97da06cb99-dbus\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-cni-binary-copy\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-netns\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-kubelet\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-conf-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.846772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-multus-certs\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.847423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrhj\" (UniqueName: \"kubernetes.io/projected/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-kube-api-access-xjrhj\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.847423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-cnibin\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.847423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.846840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-socket-dir-parent\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.847423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.847220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.848564 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.847661    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:03:49.848564 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.847749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:03:49.850084 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.849981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.853041 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.852172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:49.853558 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.853538    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.853872 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.853857    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rxm4h\""
Jul 09 18:03:49.854006 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.853857    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xtbcc\""
Jul 09 18:03:49.854075 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.854000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:03:49.854265 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.854250    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.854451 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.854263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.854540 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.854267    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.854664 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.854646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:49.854734 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:49.854718    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:03:49.855721 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.855704    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:03:49.856053 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.855941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xptgc\""
Jul 09 18:03:49.857113 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.857096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.857991 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.857973    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:03:49.859968 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.859771    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:03:49.859968 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.859787    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:03:49.859968 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.859929    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gqfpp\""
Jul 09 18:03:49.860783 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.860758    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:03:49.883272 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.883238    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6cxps"
Jul 09 18:03:49.890845 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.890729    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6cxps"
Jul 09 18:03:49.935845 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.935824    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:03:49.947589 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-etc-kubernetes\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.947686 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysctl-d\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.947686 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-sys\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.947686 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-device-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:49.947686 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.947686 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-var-lib-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.947835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-etc-kubernetes\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.947835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-modprobe-d\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.947835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysctl-conf\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.947835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-cni-bin\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.947835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-cni-multus\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-cni-bin\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-kubelet\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-cni-multus\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dde90adf-88d2-4060-8384-205e8325b6b7-agent-certs\") pod \"konnectivity-agent-9rrpv\" (UID: \"dde90adf-88d2-4060-8384-205e8325b6b7\") " pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:49.947909    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-run-netns\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-socket-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-var-lib-kubelet\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-netns\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:49.947974    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret podName:50672b24-2d0d-4afc-b7de-ac97da06cb99 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:50.447953079 +0000 UTC m=+2.897535481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret") pod "global-pull-secret-syncer-j6fjk" (UID: "50672b24-2d0d-4afc-b7de-ac97da06cb99") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.947990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-netns\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948013 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-slash\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-cni-bin\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-host\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-sys-fs\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rr2\" (UniqueName: \"kubernetes.io/projected/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-kube-api-access-k4rr2\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948207    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-tuned\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqk6w\" (UniqueName: \"kubernetes.io/projected/20a6402d-7d33-4e8b-875a-02afdf64090b-kube-api-access-nqk6w\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5696ada7-3869-49f9-b1fb-c7ad87198205-ovn-node-metrics-cert\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-systemd\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-hostroot\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-node-log\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-hostroot\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e98356-5550-4fc1-916c-bd36a3283bd8-tmp\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.948560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbgfq\" (UniqueName: \"kubernetes.io/projected/60e98356-5550-4fc1-916c-bd36a3283bd8-kube-api-access-rbgfq\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-kubelet\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-registration-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-cni-binary-copy\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-multus-certs\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-ovn\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-run\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-multus-certs\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-socket-dir-parent\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-k8s-cni-cncf-io\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-socket-dir-parent\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-ovnkube-script-lib\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.949304 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.948962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-host-run-k8s-cni-cncf-io\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysconfig\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-os-release\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-daemon-config\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-systemd-units\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nhm\" (UniqueName: \"kubernetes.io/projected/9434adf9-7671-4b69-a142-438262daa264-kube-api-access-84nhm\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thpbk\" (UniqueName: \"kubernetes.io/projected/5696ada7-3869-49f9-b1fb-c7ad87198205-kube-api-access-thpbk\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2g5\" (UniqueName: \"kubernetes.io/projected/5c353489-1b68-489f-ace9-2e463f42d74b-kube-api-access-bq2g5\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-cni-binary-copy\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-cni-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-os-release\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-conf-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-cni-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dde90adf-88d2-4060-8384-205e8325b6b7-konnectivity-ca\") pod \"konnectivity-agent-9rrpv\" (UID: \"dde90adf-88d2-4060-8384-205e8325b6b7\") " pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-ovnkube-config\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-conf-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-lib-modules\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.950059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50672b24-2d0d-4afc-b7de-ac97da06cb99-kubelet-config\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrhj\" (UniqueName: \"kubernetes.io/projected/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-kube-api-access-xjrhj\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-run-ovn-kubernetes\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-kubernetes\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50672b24-2d0d-4afc-b7de-ac97da06cb99-kubelet-config\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkm9f\" (UniqueName: \"kubernetes.io/projected/ed6e668b-385e-4854-90ce-179e2577ced6-kube-api-access-fkm9f\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-var-lib-kubelet\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-multus-daemon-config\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c353489-1b68-489f-ace9-2e463f42d74b-host\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-system-cni-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-os-release\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c353489-1b68-489f-ace9-2e463f42d74b-host\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-log-socket\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-env-overrides\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.950538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.949981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-system-cni-dir\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a6402d-7d33-4e8b-875a-02afdf64090b-tmp-dir\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c353489-1b68-489f-ace9-2e463f42d74b-serviceca\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-etc-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qvz\" (UniqueName: \"kubernetes.io/projected/937aa4bf-82d0-4e87-8755-3b8e87055e9f-kube-api-access-b2qvz\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-iptables-alerter-script\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dde90adf-88d2-4060-8384-205e8325b6b7-konnectivity-ca\") pod \"konnectivity-agent-9rrpv\" (UID: \"dde90adf-88d2-4060-8384-205e8325b6b7\") " pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50672b24-2d0d-4afc-b7de-ac97da06cb99-dbus\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-systemd\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-cni-netd\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cnibin\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-host-slash\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20a6402d-7d33-4e8b-875a-02afdf64090b-hosts-file\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50672b24-2d0d-4afc-b7de-ac97da06cb99-dbus\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-cnibin\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-cnibin\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.951014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.950638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c353489-1b68-489f-ace9-2e463f42d74b-serviceca\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.951799 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.951711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dde90adf-88d2-4060-8384-205e8325b6b7-agent-certs\") pod \"konnectivity-agent-9rrpv\" (UID: \"dde90adf-88d2-4060-8384-205e8325b6b7\") " pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:03:49.965744 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.965724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrhj\" (UniqueName: \"kubernetes.io/projected/ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38-kube-api-access-xjrhj\") pod \"multus-gvpv7\" (UID: \"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38\") " pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:49.965834 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.965772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2g5\" (UniqueName: \"kubernetes.io/projected/5c353489-1b68-489f-ace9-2e463f42d74b-kube-api-access-bq2g5\") pod \"node-ca-msphn\" (UID: \"5c353489-1b68-489f-ace9-2e463f42d74b\") " pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:49.972840 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.972797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal" event={"ID":"9722980453de5610e5dce33afc71c5c6","Type":"ContainerStarted","Data":"1f30aa31454e4875bc36dbed55ef5e237eb5214c6916a7ed819153034dd5d196"}
Jul 09 18:03:49.973917 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:49.973884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal" event={"ID":"6779a578b4eee4a48a9308987768e0e1","Type":"ContainerStarted","Data":"f912cf94c782f89bb778c5f43fa2db123343fdd5c9365752f18dd553ee193a8f"}
Jul 09 18:03:50.050820 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-systemd-units\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.050820 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nhm\" (UniqueName: \"kubernetes.io/projected/9434adf9-7671-4b69-a142-438262daa264-kube-api-access-84nhm\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thpbk\" (UniqueName: \"kubernetes.io/projected/5696ada7-3869-49f9-b1fb-c7ad87198205-kube-api-access-thpbk\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-systemd-units\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-ovnkube-config\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-lib-modules\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-run-ovn-kubernetes\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-kubernetes\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.050986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkm9f\" (UniqueName: \"kubernetes.io/projected/ed6e668b-385e-4854-90ce-179e2577ced6-kube-api-access-fkm9f\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-var-lib-kubelet\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.051015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-log-socket\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-env-overrides\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-kubernetes\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a6402d-7d33-4e8b-875a-02afdf64090b-tmp-dir\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-etc-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qvz\" (UniqueName: \"kubernetes.io/projected/937aa4bf-82d0-4e87-8755-3b8e87055e9f-kube-api-access-b2qvz\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-iptables-alerter-script\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-systemd\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-cni-netd\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cnibin\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-host-slash\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20a6402d-7d33-4e8b-875a-02afdf64090b-hosts-file\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysctl-d\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-sys\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.051527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-lib-modules\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-device-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-device-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-var-lib-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-modprobe-d\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysctl-conf\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-run-netns\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-modprobe-d\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-socket-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-slash\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-cni-bin\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-host\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-sys-fs\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-ovnkube-config\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.052435 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rr2\" (UniqueName: \"kubernetes.io/projected/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-kube-api-access-k4rr2\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-socket-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-run-ovn-kubernetes\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-sys-fs\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-var-lib-kubelet\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-cni-bin\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-host\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cnibin\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-systemd\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-host-slash\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-cni-netd\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-sys\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.051979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-tuned\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysctl-conf\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.053293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-run-netns\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20a6402d-7d33-4e8b-875a-02afdf64090b-hosts-file\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqk6w\" (UniqueName: \"kubernetes.io/projected/20a6402d-7d33-4e8b-875a-02afdf64090b-kube-api-access-nqk6w\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysctl-d\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5696ada7-3869-49f9-b1fb-c7ad87198205-ovn-node-metrics-cert\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-systemd\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-node-log\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-node-log\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.052212    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.052250    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs podName:9434adf9-7671-4b69-a142-438262daa264 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:50.552238472 +0000 UTC m=+3.001820873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs") pod "network-metrics-daemon-p55kt" (UID: "9434adf9-7671-4b69-a142-438262daa264") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-iptables-alerter-script\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-slash\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-systemd\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e98356-5550-4fc1-916c-bd36a3283bd8-tmp\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbgfq\" (UniqueName: \"kubernetes.io/projected/60e98356-5550-4fc1-916c-bd36a3283bd8-kube-api-access-rbgfq\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.054157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-kubelet\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-registration-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-ovn\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-run\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-var-lib-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-env-overrides\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-ovnkube-script-lib\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-etc-openvswitch\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-log-socket\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-host-kubelet\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5696ada7-3869-49f9-b1fb-c7ad87198205-ovnkube-script-lib\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysconfig\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.055004 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-os-release\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/937aa4bf-82d0-4e87-8755-3b8e87055e9f-os-release\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed6e668b-385e-4854-90ce-179e2577ced6-registration-dir\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5696ada7-3869-49f9-b1fb-c7ad87198205-run-ovn\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-run\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.052710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a6402d-7d33-4e8b-875a-02afdf64090b-tmp-dir\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-sysconfig\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.053916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.054041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/937aa4bf-82d0-4e87-8755-3b8e87055e9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.054683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60e98356-5550-4fc1-916c-bd36a3283bd8-etc-tuned\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.055809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.055304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5696ada7-3869-49f9-b1fb-c7ad87198205-ovn-node-metrics-cert\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.056166 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.055850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e98356-5550-4fc1-916c-bd36a3283bd8-tmp\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.070131 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.070112    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:50.070237 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.070136    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:50.070237 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.070146    2563 projected.go:196] Error preparing data for projected volume kube-api-access-266zw for pod openshift-network-diagnostics/network-check-target-ns4p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:50.070237 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.070197    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw podName:c5affad4-9ffa-40cb-8a15-ff9c327859a0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:50.570183786 +0000 UTC m=+3.019766204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-266zw" (UniqueName: "kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw") pod "network-check-target-ns4p4" (UID: "c5affad4-9ffa-40cb-8a15-ff9c327859a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:50.072916 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.072870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nhm\" (UniqueName: \"kubernetes.io/projected/9434adf9-7671-4b69-a142-438262daa264-kube-api-access-84nhm\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:50.073006 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.072913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rr2\" (UniqueName: \"kubernetes.io/projected/70d4383f-8db0-40d3-80cb-2a614fbcd8b0-kube-api-access-k4rr2\") pod \"iptables-alerter-cpmn4\" (UID: \"70d4383f-8db0-40d3-80cb-2a614fbcd8b0\") " pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:50.073829 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.073796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkm9f\" (UniqueName: \"kubernetes.io/projected/ed6e668b-385e-4854-90ce-179e2577ced6-kube-api-access-fkm9f\") pod \"aws-ebs-csi-driver-node-mn7r6\" (UID: \"ed6e668b-385e-4854-90ce-179e2577ced6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.073984 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.073961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thpbk\" (UniqueName: \"kubernetes.io/projected/5696ada7-3869-49f9-b1fb-c7ad87198205-kube-api-access-thpbk\") pod \"ovnkube-node-p5dg8\" (UID: \"5696ada7-3869-49f9-b1fb-c7ad87198205\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.074779 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.074758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qvz\" (UniqueName: \"kubernetes.io/projected/937aa4bf-82d0-4e87-8755-3b8e87055e9f-kube-api-access-b2qvz\") pod \"multus-additional-cni-plugins-zl75c\" (UID: \"937aa4bf-82d0-4e87-8755-3b8e87055e9f\") " pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.075100 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.075078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqk6w\" (UniqueName: \"kubernetes.io/projected/20a6402d-7d33-4e8b-875a-02afdf64090b-kube-api-access-nqk6w\") pod \"node-resolver-rdjmn\" (UID: \"20a6402d-7d33-4e8b-875a-02afdf64090b\") " pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:50.075162 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.075078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbgfq\" (UniqueName: \"kubernetes.io/projected/60e98356-5550-4fc1-916c-bd36a3283bd8-kube-api-access-rbgfq\") pod \"tuned-hzhjh\" (UID: \"60e98356-5550-4fc1-916c-bd36a3283bd8\") " pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.149746 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.149708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:03:50.156148 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.156126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-msphn"
Jul 09 18:03:50.166852 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.166831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gvpv7"
Jul 09 18:03:50.172586 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.172567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6"
Jul 09 18:03:50.180283 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.180265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:03:50.189036 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.189015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cpmn4"
Jul 09 18:03:50.197060 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.197044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hzhjh"
Jul 09 18:03:50.204600 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.204583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rdjmn"
Jul 09 18:03:50.211151 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.211135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zl75c"
Jul 09 18:03:50.455796 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.455757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:50.455961 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.455893    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:50.456006 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.455958    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret podName:50672b24-2d0d-4afc-b7de-ac97da06cb99 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:51.455940581 +0000 UTC m=+3.905522984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret") pod "global-pull-secret-syncer-j6fjk" (UID: "50672b24-2d0d-4afc-b7de-ac97da06cb99") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:50.557049 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.557024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:50.557247 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.557163    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:50.557247 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.557233    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs podName:9434adf9-7671-4b69-a142-438262daa264 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:51.557214427 +0000 UTC m=+4.006796848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs") pod "network-metrics-daemon-p55kt" (UID: "9434adf9-7671-4b69-a142-438262daa264") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:50.658044 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.658009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:50.658191 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.658134    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:50.658191 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.658154    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:50.658191 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.658164    2563 projected.go:196] Error preparing data for projected volume kube-api-access-266zw for pod openshift-network-diagnostics/network-check-target-ns4p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:50.658304 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:50.658213    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw podName:c5affad4-9ffa-40cb-8a15-ff9c327859a0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:51.658198177 +0000 UTC m=+4.107780594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-266zw" (UniqueName: "kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw") pod "network-check-target-ns4p4" (UID: "c5affad4-9ffa-40cb-8a15-ff9c327859a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:50.894005 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.893908    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:58:49 +0000 UTC" deadline="2028-01-15 19:26:24.630381951 +0000 UTC"
Jul 09 18:03:50.894005 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.893962    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13321h22m33.736423591s"
Jul 09 18:03:50.955007 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:50.954766    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c353489_1b68_489f_ace9_2e463f42d74b.slice/crio-c659b9437f55e94cbb3158f4170f74d908a1f4957ca23ce59f18c7f7b46ecc7b WatchSource:0}: Error finding container c659b9437f55e94cbb3158f4170f74d908a1f4957ca23ce59f18c7f7b46ecc7b: Status 404 returned error can't find the container with id c659b9437f55e94cbb3158f4170f74d908a1f4957ca23ce59f18c7f7b46ecc7b
Jul 09 18:03:50.956480 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:50.956458    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e98356_5550_4fc1_916c_bd36a3283bd8.slice/crio-3f15433e71bb733bcef4ee7d42a29e0ede2a81b30b6b87f189aff51d462c6e6f WatchSource:0}: Error finding container 3f15433e71bb733bcef4ee7d42a29e0ede2a81b30b6b87f189aff51d462c6e6f: Status 404 returned error can't find the container with id 3f15433e71bb733bcef4ee7d42a29e0ede2a81b30b6b87f189aff51d462c6e6f
Jul 09 18:03:50.959849 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:50.959485    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5696ada7_3869_49f9_b1fb_c7ad87198205.slice/crio-69fb0e4a5a6c931ca549aebab7154280abcd6a7b3d2e64363d98dc23ddeffb37 WatchSource:0}: Error finding container 69fb0e4a5a6c931ca549aebab7154280abcd6a7b3d2e64363d98dc23ddeffb37: Status 404 returned error can't find the container with id 69fb0e4a5a6c931ca549aebab7154280abcd6a7b3d2e64363d98dc23ddeffb37
Jul 09 18:03:50.962606 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:50.962579    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8f5fb4_6756_4ddf_9e3d_a60cdcc86a38.slice/crio-c212a8d7b7c90481a239168c8e671518040386857477df9c6b0969e051b0c630 WatchSource:0}: Error finding container c212a8d7b7c90481a239168c8e671518040386857477df9c6b0969e051b0c630: Status 404 returned error can't find the container with id c212a8d7b7c90481a239168c8e671518040386857477df9c6b0969e051b0c630
Jul 09 18:03:50.964367 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:03:50.963933    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a6402d_7d33_4e8b_875a_02afdf64090b.slice/crio-36384d6a38df9c77126197e963cda842c4a57a79d5a336ebb6e49ec9f0acbf71 WatchSource:0}: Error finding container 36384d6a38df9c77126197e963cda842c4a57a79d5a336ebb6e49ec9f0acbf71: Status 404 returned error can't find the container with id 36384d6a38df9c77126197e963cda842c4a57a79d5a336ebb6e49ec9f0acbf71
Jul 09 18:03:50.976703 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.976676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"69fb0e4a5a6c931ca549aebab7154280abcd6a7b3d2e64363d98dc23ddeffb37"}
Jul 09 18:03:50.977941 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.977920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hzhjh" event={"ID":"60e98356-5550-4fc1-916c-bd36a3283bd8","Type":"ContainerStarted","Data":"3f15433e71bb733bcef4ee7d42a29e0ede2a81b30b6b87f189aff51d462c6e6f"}
Jul 09 18:03:50.979102 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.979077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cpmn4" event={"ID":"70d4383f-8db0-40d3-80cb-2a614fbcd8b0","Type":"ContainerStarted","Data":"20a95f10ed854ab20b637dfb7039c5958b6fc18b20f75f6140d6cc1a2d63e41d"}
Jul 09 18:03:50.980325 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.980261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6" event={"ID":"ed6e668b-385e-4854-90ce-179e2577ced6","Type":"ContainerStarted","Data":"b3bebf5116c2d8ee889814c0ba16544c7a5ce8ab6c600fac8f6f8fc10a2c9d07"}
Jul 09 18:03:50.981376 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.981323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rdjmn" event={"ID":"20a6402d-7d33-4e8b-875a-02afdf64090b","Type":"ContainerStarted","Data":"36384d6a38df9c77126197e963cda842c4a57a79d5a336ebb6e49ec9f0acbf71"}
Jul 09 18:03:50.982867 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.982830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gvpv7" event={"ID":"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38","Type":"ContainerStarted","Data":"c212a8d7b7c90481a239168c8e671518040386857477df9c6b0969e051b0c630"}
Jul 09 18:03:50.984039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.983996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-msphn" event={"ID":"5c353489-1b68-489f-ace9-2e463f42d74b","Type":"ContainerStarted","Data":"c659b9437f55e94cbb3158f4170f74d908a1f4957ca23ce59f18c7f7b46ecc7b"}
Jul 09 18:03:50.985367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.985301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zl75c" event={"ID":"937aa4bf-82d0-4e87-8755-3b8e87055e9f","Type":"ContainerStarted","Data":"b3d424e618351f1a91e3362b15237bc6ab093e89164a9f154ba76685ebe6d900"}
Jul 09 18:03:50.986651 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:50.986632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rrpv" event={"ID":"dde90adf-88d2-4060-8384-205e8325b6b7","Type":"ContainerStarted","Data":"aad7b5c9f25e6383f0aa7edcc5a56fdaa6e7bd8048c1b2f5b5c0b10eb7a074fa"}
Jul 09 18:03:51.400274 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.400190    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:51.464591 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.464559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:51.464760 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.464697    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:51.464760 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.464757    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret podName:50672b24-2d0d-4afc-b7de-ac97da06cb99 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:53.46473934 +0000 UTC m=+5.914321745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret") pod "global-pull-secret-syncer-j6fjk" (UID: "50672b24-2d0d-4afc-b7de-ac97da06cb99") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:51.584300 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.583742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:51.584300 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.583874    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:51.584300 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.583934    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs podName:9434adf9-7671-4b69-a142-438262daa264 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:53.583916704 +0000 UTC m=+6.033499111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs") pod "network-metrics-daemon-p55kt" (UID: "9434adf9-7671-4b69-a142-438262daa264") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:51.684962 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.684926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:51.685130 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.685086    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:51.685130 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.685108    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:51.685130 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.685121    2563 projected.go:196] Error preparing data for projected volume kube-api-access-266zw for pod openshift-network-diagnostics/network-check-target-ns4p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:51.685280 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.685183    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw podName:c5affad4-9ffa-40cb-8a15-ff9c327859a0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:53.685161709 +0000 UTC m=+6.134744132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-266zw" (UniqueName: "kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw") pod "network-check-target-ns4p4" (UID: "c5affad4-9ffa-40cb-8a15-ff9c327859a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:51.894630 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.894524    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:58:49 +0000 UTC" deadline="2027-12-14 15:41:35.154306494 +0000 UTC"
Jul 09 18:03:51.894630 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.894564    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12549h37m43.259747038s"
Jul 09 18:03:51.967351 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.967239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:51.967583 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.967394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:03:51.967836 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.967816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:51.967943 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.967922    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:03:51.968003 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:51.967996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:51.968115 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:51.968066    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:03:52.003236 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:52.003201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal" event={"ID":"9722980453de5610e5dce33afc71c5c6","Type":"ContainerStarted","Data":"bc310dff5ac98a24a1f4ba62e8356876a70ed335449207369f8bbff63e92bec9"}
Jul 09 18:03:52.011845 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:52.011320    2563 generic.go:356] "Generic (PLEG): container finished" podID="6779a578b4eee4a48a9308987768e0e1" containerID="a00d22f2c7a758d94d8861441103f6aed95fe9189a4ffd9b1dfb69fb5185985a" exitCode=0
Jul 09 18:03:52.011845 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:52.011367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal" event={"ID":"6779a578b4eee4a48a9308987768e0e1","Type":"ContainerDied","Data":"a00d22f2c7a758d94d8861441103f6aed95fe9189a4ffd9b1dfb69fb5185985a"}
Jul 09 18:03:52.038602 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:52.038003    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-76.ec2.internal" podStartSLOduration=4.03798502 podStartE2EDuration="4.03798502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:52.018916494 +0000 UTC m=+4.468498918" watchObservedRunningTime="2026-07-09 18:03:52.03798502 +0000 UTC m=+4.487567444"
Jul 09 18:03:53.042796 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:53.042663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal" event={"ID":"6779a578b4eee4a48a9308987768e0e1","Type":"ContainerStarted","Data":"d35320c9317ff4c9cd67c1fdad620375471d2454b8ca4a275bc526da32876065"}
Jul 09 18:03:53.058444 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:53.058377    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-76.ec2.internal" podStartSLOduration=5.058356042 podStartE2EDuration="5.058356042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:53.057825566 +0000 UTC m=+5.507407990" watchObservedRunningTime="2026-07-09 18:03:53.058356042 +0000 UTC m=+5.507938466"
Jul 09 18:03:53.505071 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:53.504047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:53.505071 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.504287    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:53.505071 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.504350    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret podName:50672b24-2d0d-4afc-b7de-ac97da06cb99 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.504332253 +0000 UTC m=+9.953914656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret") pod "global-pull-secret-syncer-j6fjk" (UID: "50672b24-2d0d-4afc-b7de-ac97da06cb99") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:53.605343 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:53.605236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:53.605527 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.605424    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:53.605527 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.605493    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs podName:9434adf9-7671-4b69-a142-438262daa264 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.605473392 +0000 UTC m=+10.055055800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs") pod "network-metrics-daemon-p55kt" (UID: "9434adf9-7671-4b69-a142-438262daa264") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:53.705992 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:53.705953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:53.706208 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.706126    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:53.706208 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.706146    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:53.706208 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.706159    2563 projected.go:196] Error preparing data for projected volume kube-api-access-266zw for pod openshift-network-diagnostics/network-check-target-ns4p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:53.706431 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.706218    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw podName:c5affad4-9ffa-40cb-8a15-ff9c327859a0 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.706200133 +0000 UTC m=+10.155782551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-266zw" (UniqueName: "kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw") pod "network-check-target-ns4p4" (UID: "c5affad4-9ffa-40cb-8a15-ff9c327859a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:53.969565 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:53.969533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:53.969791 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.969673    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:03:53.970061 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:53.970041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:53.970160 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.970140    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:03:53.970216 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:53.970197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:53.970282 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:53.970265    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:03:55.966680 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:55.966638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:55.966680 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:55.966667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:55.967185 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:55.966795    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:03:55.967185 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:55.967016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:55.967185 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:55.967130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:03:55.967344 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:55.967214    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:03:57.540018 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:57.539963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:57.540519 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.540108    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:57.540519 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.540168    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret podName:50672b24-2d0d-4afc-b7de-ac97da06cb99 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.540151293 +0000 UTC m=+17.989733700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret") pod "global-pull-secret-syncer-j6fjk" (UID: "50672b24-2d0d-4afc-b7de-ac97da06cb99") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:57.640657 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:57.640618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:57.640822 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.640778    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:57.640884 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.640853    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs podName:9434adf9-7671-4b69-a142-438262daa264 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.640834141 +0000 UTC m=+18.090416564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs") pod "network-metrics-daemon-p55kt" (UID: "9434adf9-7671-4b69-a142-438262daa264") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:57.741903 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:57.741860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:57.742085 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.742051    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:57.742085 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.742082    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:57.742216 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.742096    2563 projected.go:196] Error preparing data for projected volume kube-api-access-266zw for pod openshift-network-diagnostics/network-check-target-ns4p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:57.742216 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.742161    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw podName:c5affad4-9ffa-40cb-8a15-ff9c327859a0 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:05.742142241 +0000 UTC m=+18.191724656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-266zw" (UniqueName: "kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw") pod "network-check-target-ns4p4" (UID: "c5affad4-9ffa-40cb-8a15-ff9c327859a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:57.967830 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:57.967795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:57.967994 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:57.967937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:57.967994 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.967943    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:03:57.968116 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.968005    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:03:57.968116 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:57.968056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:57.968218 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:57.968130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:03:59.967161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:59.967054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:03:59.967161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:59.967054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:03:59.967664 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:59.967191    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:03:59.967664 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:59.967266    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:03:59.967664 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:03:59.967326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:03:59.967664 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:03:59.967394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:01.967542 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:01.967506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:01.967997 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:01.967647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:01.967997 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:01.967665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:01.967997 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:01.967667    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:01.967997 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:01.967720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:01.967997 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:01.967765    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:03.967146 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:03.967098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:03.967146 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:03.967132    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:03.967673 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:03.967098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:03.967673 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:03.967232    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:03.967673 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:03.967322    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:03.967673 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:03.967426    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:05.601099 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:05.601058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:05.601638 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.601222    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:05.601638 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.601299    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret podName:50672b24-2d0d-4afc-b7de-ac97da06cb99 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.601283676 +0000 UTC m=+34.050866078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret") pod "global-pull-secret-syncer-j6fjk" (UID: "50672b24-2d0d-4afc-b7de-ac97da06cb99") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:05.701717 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:05.701677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:05.701891 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.701848    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:05.701954 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.701926    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs podName:9434adf9-7671-4b69-a142-438262daa264 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.701905076 +0000 UTC m=+34.151487481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs") pod "network-metrics-daemon-p55kt" (UID: "9434adf9-7671-4b69-a142-438262daa264") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:05.802779 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:05.802742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:05.802948 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.802926    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:05.803009 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.802957    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:05.803009 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.802971    2563 projected.go:196] Error preparing data for projected volume kube-api-access-266zw for pod openshift-network-diagnostics/network-check-target-ns4p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:05.803094 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.803036    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw podName:c5affad4-9ffa-40cb-8a15-ff9c327859a0 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.803017659 +0000 UTC m=+34.252600081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-266zw" (UniqueName: "kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw") pod "network-check-target-ns4p4" (UID: "c5affad4-9ffa-40cb-8a15-ff9c327859a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:05.967094 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:05.967059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:05.967297 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:05.967059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:05.967297 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.967207    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:05.967297 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:05.967059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:05.967297 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.967274    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:05.967600 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:05.967366    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:07.967587 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:07.967554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:07.968005 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:07.967641    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:07.968005 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:07.967727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:07.968005 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:07.967849    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:07.968005 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:07.967868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:07.968005 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:07.967968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:09.967778 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:09.967599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:09.968206 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:09.967599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:09.968206 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:09.967869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:09.968206 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:09.967955    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:09.968206 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:09.967672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:09.968206 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:09.968061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:10.081855 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.081830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rdjmn" event={"ID":"20a6402d-7d33-4e8b-875a-02afdf64090b","Type":"ContainerStarted","Data":"2236215957e69853a83ebe00c2b614ed20e3d4d4830b7b9d50b146194231e211"}
Jul 09 18:04:10.083204 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.083182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gvpv7" event={"ID":"ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38","Type":"ContainerStarted","Data":"0b3399865d3afa54234784475c33b2ba4d4aef6b2d21e33d27b39d10b7720f0c"}
Jul 09 18:04:10.084614 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.084593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-msphn" event={"ID":"5c353489-1b68-489f-ace9-2e463f42d74b","Type":"ContainerStarted","Data":"363b01b733c33af5c1b94c53ac572ebbe2db09852c68375a4b178a55d99044f9"}
Jul 09 18:04:10.086187 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.086159    2563 generic.go:356] "Generic (PLEG): container finished" podID="937aa4bf-82d0-4e87-8755-3b8e87055e9f" containerID="1df554c497fa4b86f7ff11756a4ff16133fc17eae5bd73a2c78ee12b98b5e051" exitCode=0
Jul 09 18:04:10.086278 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.086193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zl75c" event={"ID":"937aa4bf-82d0-4e87-8755-3b8e87055e9f","Type":"ContainerDied","Data":"1df554c497fa4b86f7ff11756a4ff16133fc17eae5bd73a2c78ee12b98b5e051"}
Jul 09 18:04:10.087825 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.087804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rrpv" event={"ID":"dde90adf-88d2-4060-8384-205e8325b6b7","Type":"ContainerStarted","Data":"507d849cb138db58cba55825044d42e0905c3d1430a252a62e40a8bce1ceebfa"}
Jul 09 18:04:10.090125 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.090109    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:04:10.090395 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.090379    2563 generic.go:356] "Generic (PLEG): container finished" podID="5696ada7-3869-49f9-b1fb-c7ad87198205" containerID="513359df86eb87366de87944f1f899346fb27c881ea6b8a077559854795e8295" exitCode=1
Jul 09 18:04:10.090486 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.090426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"c51f4e0f8697f41102726c03be87285ad10278be7ac23342dca342e9873f5d4d"}
Jul 09 18:04:10.090486 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.090478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"d279255566b9af927b6c197784fa72a1ddc964a1fb9409c0ff9f227de29bafbf"}
Jul 09 18:04:10.090571 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.090514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"3810e1135c870f1e0bd2fbd960d7ec18ddc534ff794c81de95b9a2d06727ec4e"}
Jul 09 18:04:10.090571 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.090542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerDied","Data":"513359df86eb87366de87944f1f899346fb27c881ea6b8a077559854795e8295"}
Jul 09 18:04:10.090637 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.090570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"835cd442ee5a6fae9fb9146ea5081032f9613b73ca8c2cded192843e8486c04e"}
Jul 09 18:04:10.091704 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.091686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hzhjh" event={"ID":"60e98356-5550-4fc1-916c-bd36a3283bd8","Type":"ContainerStarted","Data":"a0af34c4d6f41ab4127bccf6078f38c8c18b20099ab4e31875ddca04c558081b"}
Jul 09 18:04:10.093330 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.093311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6" event={"ID":"ed6e668b-385e-4854-90ce-179e2577ced6","Type":"ContainerStarted","Data":"2a82a8e16099cbf68065bc41674e54f923f580a00397c9140b73b620615a884e"}
Jul 09 18:04:10.098908 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.098875    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rdjmn" podStartSLOduration=3.642985687 podStartE2EDuration="22.098863879s" totalImagesPullingTime="18.455878192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:10.098655839 +0000 UTC m=+22.548238261" watchObservedRunningTime="2026-07-09 18:04:10.098863879 +0000 UTC m=+22.548446302"
Jul 09 18:04:10.133095 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.133057    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-msphn" podStartSLOduration=4.667590752 podStartE2EDuration="23.133044828s" totalImagesPullingTime="18.465454076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:10.113849392 +0000 UTC m=+22.563431814" watchObservedRunningTime="2026-07-09 18:04:10.133044828 +0000 UTC m=+22.582627295"
Jul 09 18:04:10.133302 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.133175    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gvpv7" podStartSLOduration=3.621984304 podStartE2EDuration="22.133169889s" totalImagesPullingTime="18.511185585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:10.13235186 +0000 UTC m=+22.581934317" watchObservedRunningTime="2026-07-09 18:04:10.133169889 +0000 UTC m=+22.582752311"
Jul 09 18:04:10.151246 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.151164    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hzhjh" podStartSLOduration=3.681726757 podStartE2EDuration="22.151151761s" totalImagesPullingTime="18.469425004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:10.150674917 +0000 UTC m=+22.600257339" watchObservedRunningTime="2026-07-09 18:04:10.151151761 +0000 UTC m=+22.600734184"
Jul 09 18:04:10.192179 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:10.192140    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9rrpv" podStartSLOduration=9.146466782 podStartE2EDuration="23.19212916s" totalImagesPullingTime="14.045662378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:10.191741914 +0000 UTC m=+22.641324337" watchObservedRunningTime="2026-07-09 18:04:10.19212916 +0000 UTC m=+22.641711583"
Jul 09 18:04:11.098293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.098268    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:04:11.098745 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.098698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"394877a593426883ea14baa6d80d65f4750ebe55739966c8fb3acc871453e2b6"}
Jul 09 18:04:11.100218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.100194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cpmn4" event={"ID":"70d4383f-8db0-40d3-80cb-2a614fbcd8b0","Type":"ContainerStarted","Data":"896859343af15950b4fb1df0fb8c539c83de77086552e4197b5fa5034d507a3e"}
Jul 09 18:04:11.217313 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.217287    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:04:11.911543 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.911426    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:04:11.217308557Z","UUID":"0433d980-0b92-4f74-8247-86e868626738","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:04:11.913245 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.913217    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:04:11.913367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.913262    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:04:11.967601 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.967568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:11.967781 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.967569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:11.967781 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:11.967694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:11.967781 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:11.967576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:11.967947 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:11.967870    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:11.967947 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:11.967749    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:12.105877 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:12.105716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6" event={"ID":"ed6e668b-385e-4854-90ce-179e2577ced6","Type":"ContainerStarted","Data":"b8bda478e1d36af31853ae1837297a1ca1bebea20aab80d4c416e4c11ca5bd89"}
Jul 09 18:04:12.902325 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:12.902131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:04:12.902858 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:12.902832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:04:12.920169 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:12.920121    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cpmn4" podStartSLOduration=6.500983944 podStartE2EDuration="24.920101768s" totalImagesPullingTime="18.419117824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:11.128381842 +0000 UTC m=+23.577964265" watchObservedRunningTime="2026-07-09 18:04:12.920101768 +0000 UTC m=+25.369684193"
Jul 09 18:04:13.110434 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.110396    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:04:13.110877 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.110778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"c3555c56e950ed2013afb107dfec1476bdc71dfafbbe4f9c5cec5051901c4ef2"}
Jul 09 18:04:13.112477 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.112457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6" event={"ID":"ed6e668b-385e-4854-90ce-179e2577ced6","Type":"ContainerStarted","Data":"9c8d9257ca8cff64af18f696babdc88f5ef900ac819602572dd17d6b15608701"}
Jul 09 18:04:13.112765 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.112746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:04:13.113139 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.113124    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rrpv"
Jul 09 18:04:13.131436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.131371    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mn7r6" podStartSLOduration=3.500030144 podStartE2EDuration="25.131357393s" totalImagesPullingTime="21.631327249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:13.131310153 +0000 UTC m=+25.580892577" watchObservedRunningTime="2026-07-09 18:04:13.131357393 +0000 UTC m=+25.580939815"
Jul 09 18:04:13.967051 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.967021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:13.967238 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.967021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:13.967238 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:13.967147    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:13.967367 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:13.967241    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:13.967367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:13.967020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:13.967367 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:13.967347    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:14.969943 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:14.969713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:15.093545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.093521    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:15.118358 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.118337    2563 generic.go:356] "Generic (PLEG): container finished" podID="937aa4bf-82d0-4e87-8755-3b8e87055e9f" containerID="b5e226e3497bde222c751a51d7f8739a4809f6475ddbd665a7e4362388ebe36d" exitCode=0
Jul 09 18:04:15.118464 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.118426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zl75c" event={"ID":"937aa4bf-82d0-4e87-8755-3b8e87055e9f","Type":"ContainerDied","Data":"b5e226e3497bde222c751a51d7f8739a4809f6475ddbd665a7e4362388ebe36d"}
Jul 09 18:04:15.121815 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.121801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:04:15.122215 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.122192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"5fffc32937d209d6ae34ef8fa039b64abfa140bcb8a8367d75058e8d0b3223ce"}
Jul 09 18:04:15.122612 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.122597    2563 scope.go:117] "RemoveContainer" containerID="513359df86eb87366de87944f1f899346fb27c881ea6b8a077559854795e8295"
Jul 09 18:04:15.971711 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.971631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:15.972227 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:15.971754    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:15.972227 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.971820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:15.972227 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:15.971879    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:15.972227 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:15.971932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:15.972227 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:15.972003    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:16.128027 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.128001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:04:16.128369 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.128346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" event={"ID":"5696ada7-3869-49f9-b1fb-c7ad87198205","Type":"ContainerStarted","Data":"88ea3bce1ac37500daaca7c90f39daee5e2677b48b89476e3c525b500fa9c5f8"}
Jul 09 18:04:16.128676 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.128650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:04:16.128775 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.128733    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:04:16.128834 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.128779    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:04:16.144913 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.144891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:04:16.145175 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.145156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:04:16.162524 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.162472    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8" podStartSLOduration=9.630158439 podStartE2EDuration="28.162460142s" totalImagesPullingTime="18.532301703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:16.162160638 +0000 UTC m=+28.611743062" watchObservedRunningTime="2026-07-09 18:04:16.162460142 +0000 UTC m=+28.612042565"
Jul 09 18:04:16.809616 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.809460    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ns4p4"]
Jul 09 18:04:16.809821 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.809704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:16.809821 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:16.809810    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:16.813160 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.813135    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6fjk"]
Jul 09 18:04:16.813296 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.813242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:16.813360 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:16.813341    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:16.813821 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.813800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p55kt"]
Jul 09 18:04:16.813923 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:16.813910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:16.814012 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:16.813996    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:17.132389 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:17.132309    2563 generic.go:356] "Generic (PLEG): container finished" podID="937aa4bf-82d0-4e87-8755-3b8e87055e9f" containerID="3507bf16a7cc27379d4f1f0218afb0a1b4566a554a912645c0c46fc1fae4fde0" exitCode=0
Jul 09 18:04:17.132903 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:17.132400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zl75c" event={"ID":"937aa4bf-82d0-4e87-8755-3b8e87055e9f","Type":"ContainerDied","Data":"3507bf16a7cc27379d4f1f0218afb0a1b4566a554a912645c0c46fc1fae4fde0"}
Jul 09 18:04:17.967831 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:17.967807    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:17.967998 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:17.967885    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:17.967998 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:17.967958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:17.968341 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:17.968287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:18.966771 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:18.966749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:18.967343 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:18.966849    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:19.138419 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:19.138324    2563 generic.go:356] "Generic (PLEG): container finished" podID="937aa4bf-82d0-4e87-8755-3b8e87055e9f" containerID="7da3cfac97e34ad4f036884f731184fc005663a0a199645fbade55335d6c6ca6" exitCode=0
Jul 09 18:04:19.138419 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:19.138377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zl75c" event={"ID":"937aa4bf-82d0-4e87-8755-3b8e87055e9f","Type":"ContainerDied","Data":"7da3cfac97e34ad4f036884f731184fc005663a0a199645fbade55335d6c6ca6"}
Jul 09 18:04:19.624151 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:19.624117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:19.703051 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:19.702981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:19.970362 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:19.970325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:19.970362 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:19.970354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:19.970873 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:19.970473    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:19.970873 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:19.970604    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:20.967440 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:20.967390    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:20.967613 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:20.967539    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6fjk" podUID="50672b24-2d0d-4afc-b7de-ac97da06cb99"
Jul 09 18:04:21.131584 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.131537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:21.229538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.229452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:21.385482 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.385441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:21.486294 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.486195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:21.622250 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.622209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:21.622436 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.622355    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:21.622516 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.622485    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret podName:50672b24-2d0d-4afc-b7de-ac97da06cb99 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:53.62246218 +0000 UTC m=+66.072044602 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret") pod "global-pull-secret-syncer-j6fjk" (UID: "50672b24-2d0d-4afc-b7de-ac97da06cb99") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:21.722619 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.722585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:21.722781 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.722695    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:21.722781 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.722761    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs podName:9434adf9-7671-4b69-a142-438262daa264 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:53.722746957 +0000 UTC m=+66.172329358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs") pod "network-metrics-daemon-p55kt" (UID: "9434adf9-7671-4b69-a142-438262daa264") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:21.823534 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.823451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:21.823687 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.823618    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:21.823687 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.823644    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:21.823687 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.823654    2563 projected.go:196] Error preparing data for projected volume kube-api-access-266zw for pod openshift-network-diagnostics/network-check-target-ns4p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:21.823800 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.823707    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw podName:c5affad4-9ffa-40cb-8a15-ff9c327859a0 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:53.823690525 +0000 UTC m=+66.273272929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-266zw" (UniqueName: "kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw") pod "network-check-target-ns4p4" (UID: "c5affad4-9ffa-40cb-8a15-ff9c327859a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:21.971384 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.971350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:21.971574 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.971496    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p55kt" podUID="9434adf9-7671-4b69-a142-438262daa264"
Jul 09 18:04:21.971574 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:21.971548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:21.971687 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:21.971663    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns4p4" podUID="c5affad4-9ffa-40cb-8a15-ff9c327859a0"
Jul 09 18:04:22.380557 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.380525    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-76.ec2.internal" event="NodeReady"
Jul 09 18:04:22.380968 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.380672    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:04:22.433809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.433763    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5795676c7-jvqjh"]
Jul 09 18:04:22.450488 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.450396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:22.463564 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.463539    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-psr2d"]
Jul 09 18:04:22.463726 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.463675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.466506 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.466303    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-84nts\""
Jul 09 18:04:22.466644 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.466474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:04:22.466800 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.466784    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:04:22.466869 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.466847    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:04:22.473017 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.472999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:04:22.477585 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.477563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5795676c7-jvqjh"]
Jul 09 18:04:22.477680 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.477598    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r2bq9"]
Jul 09 18:04:22.477736 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.477687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-psr2d"
Jul 09 18:04:22.481354 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.481137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:04:22.481476 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.481366    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:04:22.481646 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.481627    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:04:22.481794 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.481776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s5xfl\""
Jul 09 18:04:22.499633 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.499605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-psr2d"]
Jul 09 18:04:22.499761 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.499644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r2bq9"]
Jul 09 18:04:22.499761 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.499734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.503568 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.503540    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:04:22.504089 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.504068    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:04:22.505518 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.505481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5876s\""
Jul 09 18:04:22.601904 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.601864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:22.629496 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-image-registry-private-configuration\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.629670 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-trusted-ca\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.629670 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/111b8c89-cb2b-4293-8264-1a2307cb57ed-cert\") pod \"ingress-canary-psr2d\" (UID: \"111b8c89-cb2b-4293-8264-1a2307cb57ed\") " pod="openshift-ingress-canary/ingress-canary-psr2d"
Jul 09 18:04:22.629670 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.629670 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.629670 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-registry-certificates\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.629670 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-bound-sa-token\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.629670 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfwms\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-kube-api-access-qfwms\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.630009 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/173eee0a-fa09-40c3-9348-396facbed712-ca-trust-extracted\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.630009 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-installation-pull-secrets\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.630009 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jd8l\" (UniqueName: \"kubernetes.io/projected/111b8c89-cb2b-4293-8264-1a2307cb57ed-kube-api-access-5jd8l\") pod \"ingress-canary-psr2d\" (UID: \"111b8c89-cb2b-4293-8264-1a2307cb57ed\") " pod="openshift-ingress-canary/ingress-canary-psr2d"
Jul 09 18:04:22.630009 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111b9162-7a38-462e-bbcf-eade884b4331-tmp-dir\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.630009 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/111b9162-7a38-462e-bbcf-eade884b4331-config-volume\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.630009 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.629808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfktc\" (UniqueName: \"kubernetes.io/projected/111b9162-7a38-462e-bbcf-eade884b4331-kube-api-access-cfktc\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.730947 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.730910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-image-registry-private-configuration\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.731161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.730980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-trusted-ca\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.731161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/111b8c89-cb2b-4293-8264-1a2307cb57ed-cert\") pod \"ingress-canary-psr2d\" (UID: \"111b8c89-cb2b-4293-8264-1a2307cb57ed\") " pod="openshift-ingress-canary/ingress-canary-psr2d"
Jul 09 18:04:22.731161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.731161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.731367 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:22.731179    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:22.731367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-registry-certificates\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.731367 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:22.731252    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls podName:111b9162-7a38-462e-bbcf-eade884b4331 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:23.231231289 +0000 UTC m=+35.680813705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls") pod "dns-default-r2bq9" (UID: "111b9162-7a38-462e-bbcf-eade884b4331") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:22.731367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-bound-sa-token\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.731367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfwms\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-kube-api-access-qfwms\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.731367 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:22.731354    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:22.731367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/173eee0a-fa09-40c3-9348-396facbed712-ca-trust-extracted\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.731367 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:22.731370    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5795676c7-jvqjh: secret "image-registry-tls" not found
Jul 09 18:04:22.731782 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-installation-pull-secrets\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.731782 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jd8l\" (UniqueName: \"kubernetes.io/projected/111b8c89-cb2b-4293-8264-1a2307cb57ed-kube-api-access-5jd8l\") pod \"ingress-canary-psr2d\" (UID: \"111b8c89-cb2b-4293-8264-1a2307cb57ed\") " pod="openshift-ingress-canary/ingress-canary-psr2d"
Jul 09 18:04:22.731782 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:22.731434    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls podName:173eee0a-fa09-40c3-9348-396facbed712 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:23.23140014 +0000 UTC m=+35.680982564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls") pod "image-registry-5795676c7-jvqjh" (UID: "173eee0a-fa09-40c3-9348-396facbed712") : secret "image-registry-tls" not found
Jul 09 18:04:22.731782 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111b9162-7a38-462e-bbcf-eade884b4331-tmp-dir\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.731782 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/111b9162-7a38-462e-bbcf-eade884b4331-config-volume\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.731782 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfktc\" (UniqueName: \"kubernetes.io/projected/111b9162-7a38-462e-bbcf-eade884b4331-kube-api-access-cfktc\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.732080 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.731886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111b9162-7a38-462e-bbcf-eade884b4331-tmp-dir\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.732080 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.732035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/173eee0a-fa09-40c3-9348-396facbed712-ca-trust-extracted\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.732198 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.732173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-trusted-ca\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.732548 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.732346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-registry-certificates\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.732548 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.732355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/111b9162-7a38-462e-bbcf-eade884b4331-config-volume\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.735895 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.735869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-installation-pull-secrets\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.735895 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.735884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/111b8c89-cb2b-4293-8264-1a2307cb57ed-cert\") pod \"ingress-canary-psr2d\" (UID: \"111b8c89-cb2b-4293-8264-1a2307cb57ed\") " pod="openshift-ingress-canary/ingress-canary-psr2d"
Jul 09 18:04:22.736034 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.735902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-image-registry-private-configuration\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.750730 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.750690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfktc\" (UniqueName: \"kubernetes.io/projected/111b9162-7a38-462e-bbcf-eade884b4331-kube-api-access-cfktc\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:22.750957 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.750847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-bound-sa-token\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.751284 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.751236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jd8l\" (UniqueName: \"kubernetes.io/projected/111b8c89-cb2b-4293-8264-1a2307cb57ed-kube-api-access-5jd8l\") pod \"ingress-canary-psr2d\" (UID: \"111b8c89-cb2b-4293-8264-1a2307cb57ed\") " pod="openshift-ingress-canary/ingress-canary-psr2d"
Jul 09 18:04:22.751984 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.751958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfwms\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-kube-api-access-qfwms\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:22.760163 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.760126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:22.788394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.788348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-psr2d"
Jul 09 18:04:22.862214 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.862178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:22.967018 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.966983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:22.969704 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:22.969679    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:04:23.164068 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.163971    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"]
Jul 09 18:04:23.185291 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.185262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"]
Jul 09 18:04:23.185472 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.185418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.189148 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.189126    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:23.189287 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.189177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dn9gr\""
Jul 09 18:04:23.189839 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.189820    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 09 18:04:23.189921 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.189863    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 09 18:04:23.190154 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.190136    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:23.236015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.235972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:23.236187 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.236026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:23.236187 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:23.236128    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:23.236187 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:23.236144    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:23.236187 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:23.236156    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5795676c7-jvqjh: secret "image-registry-tls" not found
Jul 09 18:04:23.236361 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:23.236195    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls podName:111b9162-7a38-462e-bbcf-eade884b4331 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:24.236176603 +0000 UTC m=+36.685759011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls") pod "dns-default-r2bq9" (UID: "111b9162-7a38-462e-bbcf-eade884b4331") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:23.236361 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:23.236213    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls podName:173eee0a-fa09-40c3-9348-396facbed712 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:24.236204309 +0000 UTC m=+36.685786715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls") pod "image-registry-5795676c7-jvqjh" (UID: "173eee0a-fa09-40c3-9348-396facbed712") : secret "image-registry-tls" not found
Jul 09 18:04:23.337279 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.337246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01a20cc4-485d-40ff-847e-c0034eeaa8c5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.337279 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.337283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjng\" (UniqueName: \"kubernetes.io/projected/01a20cc4-485d-40ff-847e-c0034eeaa8c5-kube-api-access-bgjng\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.337486 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.337349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01a20cc4-485d-40ff-847e-c0034eeaa8c5-config\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.437771 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.437742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01a20cc4-485d-40ff-847e-c0034eeaa8c5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.438140 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.437777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjng\" (UniqueName: \"kubernetes.io/projected/01a20cc4-485d-40ff-847e-c0034eeaa8c5-kube-api-access-bgjng\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.438140 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.437804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01a20cc4-485d-40ff-847e-c0034eeaa8c5-config\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.438425 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.438390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01a20cc4-485d-40ff-847e-c0034eeaa8c5-config\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.440323 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.440294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01a20cc4-485d-40ff-847e-c0034eeaa8c5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.448007 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.447981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjng\" (UniqueName: \"kubernetes.io/projected/01a20cc4-485d-40ff-847e-c0034eeaa8c5-kube-api-access-bgjng\") pod \"service-ca-operator-7695fc6c8c-ftb9z\" (UID: \"01a20cc4-485d-40ff-847e-c0034eeaa8c5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.496687 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.496656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"
Jul 09 18:04:23.514442 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.514390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-psr2d"]
Jul 09 18:04:23.569862 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:23.569832    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111b8c89_cb2b_4293_8264_1a2307cb57ed.slice/crio-bf28444db3905fbb0729ea9083dac1c565c0db2cf07151eec42660f32c604ac6 WatchSource:0}: Error finding container bf28444db3905fbb0729ea9083dac1c565c0db2cf07151eec42660f32c604ac6: Status 404 returned error can't find the container with id bf28444db3905fbb0729ea9083dac1c565c0db2cf07151eec42660f32c604ac6
Jul 09 18:04:23.723158 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.723127    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z"]
Jul 09 18:04:23.726576 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:23.726548    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a20cc4_485d_40ff_847e_c0034eeaa8c5.slice/crio-7dcf8453a2b1e819a889c5cd39e0bade60498261456397b67f2f34d426b53712 WatchSource:0}: Error finding container 7dcf8453a2b1e819a889c5cd39e0bade60498261456397b67f2f34d426b53712: Status 404 returned error can't find the container with id 7dcf8453a2b1e819a889c5cd39e0bade60498261456397b67f2f34d426b53712
Jul 09 18:04:23.969964 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.969894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:23.970114 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.970074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:23.973605 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.973574    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jfsz2\""
Jul 09 18:04:23.973704 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.973646    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:04:23.974316 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.974303    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:04:23.974627 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.974606    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sjz5q\""
Jul 09 18:04:23.974692 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:23.974627    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:04:24.154321 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:24.154291    2563 generic.go:356] "Generic (PLEG): container finished" podID="937aa4bf-82d0-4e87-8755-3b8e87055e9f" containerID="767ca01d4c6cb3395c566142966f1e90b2c054699dd6d06fb0baed73d8cc4ae0" exitCode=0
Jul 09 18:04:24.154561 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:24.154384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zl75c" event={"ID":"937aa4bf-82d0-4e87-8755-3b8e87055e9f","Type":"ContainerDied","Data":"767ca01d4c6cb3395c566142966f1e90b2c054699dd6d06fb0baed73d8cc4ae0"}
Jul 09 18:04:24.155899 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:24.155870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-psr2d" event={"ID":"111b8c89-cb2b-4293-8264-1a2307cb57ed","Type":"ContainerStarted","Data":"bf28444db3905fbb0729ea9083dac1c565c0db2cf07151eec42660f32c604ac6"}
Jul 09 18:04:24.157159 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:24.157125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z" event={"ID":"01a20cc4-485d-40ff-847e-c0034eeaa8c5","Type":"ContainerStarted","Data":"7dcf8453a2b1e819a889c5cd39e0bade60498261456397b67f2f34d426b53712"}
Jul 09 18:04:24.243743 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:24.243719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:24.243911 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:24.243871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:24.244369 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:24.243876    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:24.244369 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:24.244052    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:24.244369 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:24.244070    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5795676c7-jvqjh: secret "image-registry-tls" not found
Jul 09 18:04:24.244369 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:24.244122    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls podName:111b9162-7a38-462e-bbcf-eade884b4331 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.244101252 +0000 UTC m=+38.693683660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls") pod "dns-default-r2bq9" (UID: "111b9162-7a38-462e-bbcf-eade884b4331") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:24.244369 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:24.244137    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls podName:173eee0a-fa09-40c3-9348-396facbed712 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.244130313 +0000 UTC m=+38.693712714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls") pod "image-registry-5795676c7-jvqjh" (UID: "173eee0a-fa09-40c3-9348-396facbed712") : secret "image-registry-tls" not found
Jul 09 18:04:24.997117 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:24.997078    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kss8w"]
Jul 09 18:04:25.000601 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.000578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.006621 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.006597    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:04:25.007340 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.007305    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:04:25.007497 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.007377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4gbrj\""
Jul 09 18:04:25.007497 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.007429    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:04:25.007605 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.007311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:04:25.017051 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.016988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kss8w"]
Jul 09 18:04:25.049472 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.049395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.049611 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.049553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/01c2b861-d2e0-459f-889b-fad5bf3abaea-crio-socket\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.049611 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.049604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/01c2b861-d2e0-459f-889b-fad5bf3abaea-data-volume\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.049716 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.049629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgww\" (UniqueName: \"kubernetes.io/projected/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-api-access-mpgww\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.049716 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.049654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.063846 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.063784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:25.148125 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.148087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:25.150000 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.149970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/01c2b861-d2e0-459f-889b-fad5bf3abaea-crio-socket\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.150114 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.150034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/01c2b861-d2e0-459f-889b-fad5bf3abaea-data-volume\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.150114 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.150060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgww\" (UniqueName: \"kubernetes.io/projected/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-api-access-mpgww\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.150114 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.150081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.150114 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.150102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.150319 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.150203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/01c2b861-d2e0-459f-889b-fad5bf3abaea-crio-socket\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.150319 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:25.150298    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:25.150431 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:25.150357    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:25.650343819 +0000 UTC m=+38.099926240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:25.150542 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:25.150508    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:25.150589 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:25.150567    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:25.650552047 +0000 UTC m=+38.100134469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:25.150638 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.150607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/01c2b861-d2e0-459f-889b-fad5bf3abaea-data-volume\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.163528 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.163505    2563 generic.go:356] "Generic (PLEG): container finished" podID="937aa4bf-82d0-4e87-8755-3b8e87055e9f" containerID="25e9d921a7ab334f0bb0aa0f672011d1e39538456faf6495cbaac59595ce05f9" exitCode=0
Jul 09 18:04:25.163623 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.163538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zl75c" event={"ID":"937aa4bf-82d0-4e87-8755-3b8e87055e9f","Type":"ContainerDied","Data":"25e9d921a7ab334f0bb0aa0f672011d1e39538456faf6495cbaac59595ce05f9"}
Jul 09 18:04:25.179235 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.179216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgww\" (UniqueName: \"kubernetes.io/projected/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-api-access-mpgww\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.653720 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.653684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.653906 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.653729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:25.653906 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:25.653822    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:25.653906 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:25.653886    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:25.654074 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:25.653951    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.653931625 +0000 UTC m=+39.103514043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:25.654074 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:25.653972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.65396181 +0000 UTC m=+39.103544219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:25.845280 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.845245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:25.967186 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:25.967131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:26.258194 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.258108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:26.258194 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.258188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:26.258821 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.258292    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:26.258821 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.258326    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:26.258821 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.258339    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5795676c7-jvqjh: secret "image-registry-tls" not found
Jul 09 18:04:26.258821 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.258364    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls podName:111b9162-7a38-462e-bbcf-eade884b4331 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:30.258343193 +0000 UTC m=+42.707925614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls") pod "dns-default-r2bq9" (UID: "111b9162-7a38-462e-bbcf-eade884b4331") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:26.258821 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.258381    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls podName:173eee0a-fa09-40c3-9348-396facbed712 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:30.258372834 +0000 UTC m=+42.707955235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls") pod "image-registry-5795676c7-jvqjh" (UID: "173eee0a-fa09-40c3-9348-396facbed712") : secret "image-registry-tls" not found
Jul 09 18:04:26.546271 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.546173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:26.579487 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.579460    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5"]
Jul 09 18:04:26.582614 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.582598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5"
Jul 09 18:04:26.585243 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.585225    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 18:04:26.586176 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.586155    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:26.586243 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.586157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qjkvv\""
Jul 09 18:04:26.594026 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.594005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5"]
Jul 09 18:04:26.644096 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.643918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:26.661024 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.661000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:26.661141 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.661041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:26.661141 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.661067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrjr\" (UniqueName: \"kubernetes.io/projected/d10396de-6fa9-4cd3-a6d4-95615a49dce8-kube-api-access-whrjr\") pod \"migrator-7f59b54f6c-g4nd5\" (UID: \"d10396de-6fa9-4cd3-a6d4-95615a49dce8\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5"
Jul 09 18:04:26.661247 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.661143    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:26.661247 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.661153    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:26.661247 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.661189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:28.661176439 +0000 UTC m=+41.110758840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:26.661247 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:26.661211    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:28.661194375 +0000 UTC m=+41.110776782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:26.745039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.744986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:26.762039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.762018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrjr\" (UniqueName: \"kubernetes.io/projected/d10396de-6fa9-4cd3-a6d4-95615a49dce8-kube-api-access-whrjr\") pod \"migrator-7f59b54f6c-g4nd5\" (UID: \"d10396de-6fa9-4cd3-a6d4-95615a49dce8\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5"
Jul 09 18:04:26.771418 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.771378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrjr\" (UniqueName: \"kubernetes.io/projected/d10396de-6fa9-4cd3-a6d4-95615a49dce8-kube-api-access-whrjr\") pod \"migrator-7f59b54f6c-g4nd5\" (UID: \"d10396de-6fa9-4cd3-a6d4-95615a49dce8\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5"
Jul 09 18:04:26.805062 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.805005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rdjmn_20a6402d-7d33-4e8b-875a-02afdf64090b/dns-node-resolver/0.log"
Jul 09 18:04:26.891693 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:26.891648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5"
Jul 09 18:04:27.048485 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.048458    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5"]
Jul 09 18:04:27.052426 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:27.052383    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10396de_6fa9_4cd3_a6d4_95615a49dce8.slice/crio-0271c463ceb4727e37dab4a4aa3a9f0e39b216b151001e5d32aadee780c84d44 WatchSource:0}: Error finding container 0271c463ceb4727e37dab4a4aa3a9f0e39b216b151001e5d32aadee780c84d44: Status 404 returned error can't find the container with id 0271c463ceb4727e37dab4a4aa3a9f0e39b216b151001e5d32aadee780c84d44
Jul 09 18:04:27.167276 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.167237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:27.169520 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.169495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-psr2d" event={"ID":"111b8c89-cb2b-4293-8264-1a2307cb57ed","Type":"ContainerStarted","Data":"cbb98adbf100bd726957ef2dae89cfb5464c39fea9fb405f0456bc3de289d31f"}
Jul 09 18:04:27.171002 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.170979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z" event={"ID":"01a20cc4-485d-40ff-847e-c0034eeaa8c5","Type":"ContainerStarted","Data":"98850aa5b6e9e925ecabb11f01fe9080a269e010cdba4cd105ae11a13bf48837"}
Jul 09 18:04:27.172134 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.172113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5" event={"ID":"d10396de-6fa9-4cd3-a6d4-95615a49dce8","Type":"ContainerStarted","Data":"0271c463ceb4727e37dab4a4aa3a9f0e39b216b151001e5d32aadee780c84d44"}
Jul 09 18:04:27.179175 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.179155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zl75c" event={"ID":"937aa4bf-82d0-4e87-8755-3b8e87055e9f","Type":"ContainerStarted","Data":"e8300c8a7d268acbd002fb3999acaf664ca57190bfcd215a4052f835f65ab028"}
Jul 09 18:04:27.188377 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.188337    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-psr2d" podStartSLOduration=2.405864749 podStartE2EDuration="5.188326675s" totalImagesPullingTime="2.782461926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:27.187454743 +0000 UTC m=+39.637037165" watchObservedRunningTime="2026-07-09 18:04:27.188326675 +0000 UTC m=+39.637909098"
Jul 09 18:04:27.216770 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.216734    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z" podStartSLOduration=1.581138373 podStartE2EDuration="4.216722524s" totalImagesPullingTime="2.635584151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:27.215918445 +0000 UTC m=+39.665500881" watchObservedRunningTime="2026-07-09 18:04:27.216722524 +0000 UTC m=+39.666304946"
Jul 09 18:04:27.242398 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.242350    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zl75c" podStartSLOduration=9.190230155 podStartE2EDuration="39.242331806s" totalImagesPullingTime="30.052101651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:27.242119424 +0000 UTC m=+39.691701847" watchObservedRunningTime="2026-07-09 18:04:27.242331806 +0000 UTC m=+39.691914230"
Jul 09 18:04:27.268680 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.268654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:27.600060 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:27.600032    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-msphn_5c353489-1b68-489f-ace9-2e463f42d74b/node-ca/0.log"
Jul 09 18:04:28.401251 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:28.401225    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-psr2d_111b8c89-cb2b-4293-8264-1a2307cb57ed/serve-healthcheck-canary/0.log"
Jul 09 18:04:28.674668 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:28.674645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:28.674785 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:28.674677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:28.674853 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:28.674817    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:28.674903 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:28.674851    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:28.674903 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:28.674879    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.674862537 +0000 UTC m=+45.124444943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:28.674995 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:28.674911    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.674893388 +0000 UTC m=+45.124475804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:29.185271 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:29.185235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5" event={"ID":"d10396de-6fa9-4cd3-a6d4-95615a49dce8","Type":"ContainerStarted","Data":"3476e2c489fde03c1350b7edcd9cb7a4c0aa1345f463bb801eaa507630b284e6"}
Jul 09 18:04:29.185460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:29.185296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5" event={"ID":"d10396de-6fa9-4cd3-a6d4-95615a49dce8","Type":"ContainerStarted","Data":"a2e34336b9db09fac612480277832b90cceb73826340b14b12e15a5f80fe8088"}
Jul 09 18:04:29.201692 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:29.201647    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-g4nd5" podStartSLOduration=1.775163084 podStartE2EDuration="3.201631444s" totalImagesPullingTime="1.42646836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.201016982 +0000 UTC m=+41.650599425" watchObservedRunningTime="2026-07-09 18:04:29.201631444 +0000 UTC m=+41.651213867"
Jul 09 18:04:30.287641 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.287604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:30.288003 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.287653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:30.288003 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:30.287754    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:30.288003 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:30.287758    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:30.288003 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:30.287829    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls podName:111b9162-7a38-462e-bbcf-eade884b4331 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:38.287808994 +0000 UTC m=+50.737391396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls") pod "dns-default-r2bq9" (UID: "111b9162-7a38-462e-bbcf-eade884b4331") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:30.288003 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:30.287764    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5795676c7-jvqjh: secret "image-registry-tls" not found
Jul 09 18:04:30.288003 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:30.287883    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls podName:173eee0a-fa09-40c3-9348-396facbed712 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:38.287872935 +0000 UTC m=+50.737455369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls") pod "image-registry-5795676c7-jvqjh" (UID: "173eee0a-fa09-40c3-9348-396facbed712") : secret "image-registry-tls" not found
Jul 09 18:04:30.325006 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.324976    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-75295"]
Jul 09 18:04:30.341243 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.341220    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-75295"]
Jul 09 18:04:30.341377 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.341312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.343826 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.343803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 18:04:30.343963 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.343838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 18:04:30.344823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.344800    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-52p8f\""
Jul 09 18:04:30.344989 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.344839    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 18:04:30.344989 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.344805    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 18:04:30.387978 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.387947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c7f7f35-382c-46b3-964b-b9b78b6f543d-signing-key\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.388163 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.388016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c7f7f35-382c-46b3-964b-b9b78b6f543d-signing-cabundle\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.388163 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.388070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82hs\" (UniqueName: \"kubernetes.io/projected/7c7f7f35-382c-46b3-964b-b9b78b6f543d-kube-api-access-n82hs\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.489129 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.489096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c7f7f35-382c-46b3-964b-b9b78b6f543d-signing-key\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.489282 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.489146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c7f7f35-382c-46b3-964b-b9b78b6f543d-signing-cabundle\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.489282 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.489266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82hs\" (UniqueName: \"kubernetes.io/projected/7c7f7f35-382c-46b3-964b-b9b78b6f543d-kube-api-access-n82hs\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.489756 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.489742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c7f7f35-382c-46b3-964b-b9b78b6f543d-signing-cabundle\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.492179 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.492159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c7f7f35-382c-46b3-964b-b9b78b6f543d-signing-key\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.497489 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.497466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82hs\" (UniqueName: \"kubernetes.io/projected/7c7f7f35-382c-46b3-964b-b9b78b6f543d-kube-api-access-n82hs\") pod \"service-ca-79f68b9f4f-75295\" (UID: \"7c7f7f35-382c-46b3-964b-b9b78b6f543d\") " pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.649967 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.649881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-75295"
Jul 09 18:04:30.789320 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:30.789284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-75295"]
Jul 09 18:04:30.792221 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:30.792192    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7f7f35_382c_46b3_964b_b9b78b6f543d.slice/crio-e7776f1d7b6c03f99b17c825090a5a904452b7a8bfa8bddb6f9c84c378034508 WatchSource:0}: Error finding container e7776f1d7b6c03f99b17c825090a5a904452b7a8bfa8bddb6f9c84c378034508: Status 404 returned error can't find the container with id e7776f1d7b6c03f99b17c825090a5a904452b7a8bfa8bddb6f9c84c378034508
Jul 09 18:04:31.191336 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:31.191301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-75295" event={"ID":"7c7f7f35-382c-46b3-964b-b9b78b6f543d","Type":"ContainerStarted","Data":"25b03b8f9b386a546c0f25d1879d0ff83b1ca19d92301bc3c24c2428d50c6043"}
Jul 09 18:04:31.191540 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:31.191349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-75295" event={"ID":"7c7f7f35-382c-46b3-964b-b9b78b6f543d","Type":"ContainerStarted","Data":"e7776f1d7b6c03f99b17c825090a5a904452b7a8bfa8bddb6f9c84c378034508"}
Jul 09 18:04:31.211133 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:31.211081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-75295" podStartSLOduration=1.211064975 podStartE2EDuration="1.211064975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:31.209970932 +0000 UTC m=+43.659553355" watchObservedRunningTime="2026-07-09 18:04:31.211064975 +0000 UTC m=+43.660647420"
Jul 09 18:04:32.706784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:32.706749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:32.707239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:32.706797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:32.707239 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:32.706918    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:32.707239 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:04:32.706999    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls podName:01c2b861-d2e0-459f-889b-fad5bf3abaea nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.706977786 +0000 UTC m=+53.156560190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kss8w" (UID: "01c2b861-d2e0-459f-889b-fad5bf3abaea") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:32.707420 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:32.707387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01c2b861-d2e0-459f-889b-fad5bf3abaea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:38.350486 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:38.350447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:38.350931 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:38.350496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:38.352702 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:38.352679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/111b9162-7a38-462e-bbcf-eade884b4331-metrics-tls\") pod \"dns-default-r2bq9\" (UID: \"111b9162-7a38-462e-bbcf-eade884b4331\") " pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:38.361757 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:38.361734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"image-registry-5795676c7-jvqjh\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") " pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:38.375525 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:38.375506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:38.410617 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:38.410595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:38.525824 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:38.525797    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5795676c7-jvqjh"]
Jul 09 18:04:38.535054 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:38.535026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173eee0a_fa09_40c3_9348_396facbed712.slice/crio-5174cfd819df4f2c4275e59d9387a359420cd4454609795854a9a8a6eaeb8625 WatchSource:0}: Error finding container 5174cfd819df4f2c4275e59d9387a359420cd4454609795854a9a8a6eaeb8625: Status 404 returned error can't find the container with id 5174cfd819df4f2c4275e59d9387a359420cd4454609795854a9a8a6eaeb8625
Jul 09 18:04:38.562652 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:38.562629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r2bq9"]
Jul 09 18:04:38.565508 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:38.565484    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111b9162_7a38_462e_bbcf_eade884b4331.slice/crio-a9bb2b864cae1a78b8c65782af723d270adfbbc7541556ece8cf22c107b2a7fd WatchSource:0}: Error finding container a9bb2b864cae1a78b8c65782af723d270adfbbc7541556ece8cf22c107b2a7fd: Status 404 returned error can't find the container with id a9bb2b864cae1a78b8c65782af723d270adfbbc7541556ece8cf22c107b2a7fd
Jul 09 18:04:39.214715 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:39.214669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r2bq9" event={"ID":"111b9162-7a38-462e-bbcf-eade884b4331","Type":"ContainerStarted","Data":"a9bb2b864cae1a78b8c65782af723d270adfbbc7541556ece8cf22c107b2a7fd"}
Jul 09 18:04:39.216107 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:39.216076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5795676c7-jvqjh" event={"ID":"173eee0a-fa09-40c3-9348-396facbed712","Type":"ContainerStarted","Data":"8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3"}
Jul 09 18:04:39.216227 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:39.216123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5795676c7-jvqjh" event={"ID":"173eee0a-fa09-40c3-9348-396facbed712","Type":"ContainerStarted","Data":"5174cfd819df4f2c4275e59d9387a359420cd4454609795854a9a8a6eaeb8625"}
Jul 09 18:04:39.216227 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:39.216166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:04:39.248584 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:39.248541    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5795676c7-jvqjh" podStartSLOduration=20.248527702 podStartE2EDuration="20.248527702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:39.246950491 +0000 UTC m=+51.696532927" watchObservedRunningTime="2026-07-09 18:04:39.248527702 +0000 UTC m=+51.698110115"
Jul 09 18:04:40.769039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:40.768967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:40.771734 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:40.771704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01c2b861-d2e0-459f-889b-fad5bf3abaea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kss8w\" (UID: \"01c2b861-d2e0-459f-889b-fad5bf3abaea\") " pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:40.913693 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:40.913652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kss8w"
Jul 09 18:04:41.188535 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:41.188461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kss8w"]
Jul 09 18:04:41.191159 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:41.191134    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c2b861_d2e0_459f_889b_fad5bf3abaea.slice/crio-7429f469f023db29b21c8478a1fd428acc092efcd9ea209d939d4b138e907d1f WatchSource:0}: Error finding container 7429f469f023db29b21c8478a1fd428acc092efcd9ea209d939d4b138e907d1f: Status 404 returned error can't find the container with id 7429f469f023db29b21c8478a1fd428acc092efcd9ea209d939d4b138e907d1f
Jul 09 18:04:41.226475 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:41.226440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kss8w" event={"ID":"01c2b861-d2e0-459f-889b-fad5bf3abaea","Type":"ContainerStarted","Data":"7429f469f023db29b21c8478a1fd428acc092efcd9ea209d939d4b138e907d1f"}
Jul 09 18:04:41.228149 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:41.228078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r2bq9" event={"ID":"111b9162-7a38-462e-bbcf-eade884b4331","Type":"ContainerStarted","Data":"8f91c6fa18b222dbb0e6273dc4ffd7eb3a7260fbd31138de7f5ad90f23631307"}
Jul 09 18:04:42.235121 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:42.235078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r2bq9" event={"ID":"111b9162-7a38-462e-bbcf-eade884b4331","Type":"ContainerStarted","Data":"6170a6e845e636e8515bddf8de476d3e5c93d546967b2e80eb8c4bb1562b05fb"}
Jul 09 18:04:42.235659 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:42.235228    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:42.261562 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:42.261364    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r2bq9" podStartSLOduration=17.739894202 podStartE2EDuration="20.261344659s" totalImagesPullingTime="2.521450457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:42.259812871 +0000 UTC m=+54.709395323" watchObservedRunningTime="2026-07-09 18:04:42.261344659 +0000 UTC m=+54.710927086"
Jul 09 18:04:45.248819 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:45.248783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kss8w" event={"ID":"01c2b861-d2e0-459f-889b-fad5bf3abaea","Type":"ContainerStarted","Data":"2fc76ac99f76287db49ba819fb9b74f685285145b9057faf11f0f96d2d10c6bd"}
Jul 09 18:04:47.258068 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:47.258025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kss8w" event={"ID":"01c2b861-d2e0-459f-889b-fad5bf3abaea","Type":"ContainerStarted","Data":"3d9fefd8970e8531ed9856785c3412a87b537f78c43135bd4c747e55deb1b9cc"}
Jul 09 18:04:48.147210 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:48.147174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5dg8"
Jul 09 18:04:49.266130 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:49.266094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kss8w" event={"ID":"01c2b861-d2e0-459f-889b-fad5bf3abaea","Type":"ContainerStarted","Data":"15ea510d528dd32bdeab644bbac358e6e3d53eda7614ddc6d40c5043924192ab"}
Jul 09 18:04:49.291807 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:49.291753    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kss8w" podStartSLOduration=17.814622772 podStartE2EDuration="25.291737296s" totalImagesPullingTime="7.477114524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:49.290593552 +0000 UTC m=+61.740175994" watchObservedRunningTime="2026-07-09 18:04:49.291737296 +0000 UTC m=+61.741319736"
Jul 09 18:04:52.241770 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:52.241742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r2bq9"
Jul 09 18:04:53.648931 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.648900    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"]
Jul 09 18:04:53.670875 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.670850    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"]
Jul 09 18:04:53.671037 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.670972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"
Jul 09 18:04:53.671375 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.671352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:53.674077 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.674056    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:04:53.674211 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.674089    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 09 18:04:53.674211 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.674100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ljfpz\""
Jul 09 18:04:53.674319 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.674221    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 09 18:04:53.683975 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.683956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50672b24-2d0d-4afc-b7de-ac97da06cb99-original-pull-secret\") pod \"global-pull-secret-syncer-j6fjk\" (UID: \"50672b24-2d0d-4afc-b7de-ac97da06cb99\") " pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:53.684811 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.684796    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"]
Jul 09 18:04:53.684947 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.684930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.690914 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.690892    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 18:04:53.691564 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.691542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 18:04:53.691665 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.691575    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 18:04:53.693301 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.693285    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 18:04:53.706918 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.706897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"]
Jul 09 18:04:53.707012 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.706938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"]
Jul 09 18:04:53.707012 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.706956    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"]
Jul 09 18:04:53.707092 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.707015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"
Jul 09 18:04:53.710380 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.710362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 18:04:53.710602 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.710583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-r87t5\""
Jul 09 18:04:53.719881 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.719861    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5795676c7-jvqjh"]
Jul 09 18:04:53.723974 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.723950    2563 patch_prober.go:28] interesting pod/image-registry-5795676c7-jvqjh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:04:53.724049 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.724000    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5795676c7-jvqjh" podUID="173eee0a-fa09-40c3-9348-396facbed712" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:04:53.753994 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.753969    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk"]
Jul 09 18:04:53.771797 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.771776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edf89f9f-5e8c-41ca-899a-2e77089e36c5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f56d866d8-frswm\" (UID: \"edf89f9f-5e8c-41ca-899a-2e77089e36c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"
Jul 09 18:04:53.771914 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.771810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f12fbd2f-6972-41b9-af69-b36fb3b29045-tmp\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.771914 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.771835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlmw\" (UniqueName: \"kubernetes.io/projected/f12fbd2f-6972-41b9-af69-b36fb3b29045-kube-api-access-6dlmw\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.771914 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.771865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:53.771914 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.771887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvj87\" (UniqueName: \"kubernetes.io/projected/edf89f9f-5e8c-41ca-899a-2e77089e36c5-kube-api-access-tvj87\") pod \"managed-serviceaccount-addon-agent-7f56d866d8-frswm\" (UID: \"edf89f9f-5e8c-41ca-899a-2e77089e36c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"
Jul 09 18:04:53.771914 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.771911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/185ed8c1-2274-4cef-b764-caa9d400a138-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-6zkf5\" (UID: \"185ed8c1-2274-4cef-b764-caa9d400a138\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"
Jul 09 18:04:53.772085 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.771935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f12fbd2f-6972-41b9-af69-b36fb3b29045-klusterlet-config\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.772085 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.771974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/185ed8c1-2274-4cef-b764-caa9d400a138-nginx-conf\") pod \"networking-console-plugin-6cf856758c-6zkf5\" (UID: \"185ed8c1-2274-4cef-b764-caa9d400a138\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"
Jul 09 18:04:53.778647 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.778629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:04:53.779431 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.779395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk"
Jul 09 18:04:53.783556 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.783534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-l5sck\""
Jul 09 18:04:53.784837 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.784817    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk"]
Jul 09 18:04:53.785232 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.785210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9434adf9-7671-4b69-a142-438262daa264-metrics-certs\") pod \"network-metrics-daemon-p55kt\" (UID: \"9434adf9-7671-4b69-a142-438262daa264\") " pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:53.872985 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.872955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/185ed8c1-2274-4cef-b764-caa9d400a138-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-6zkf5\" (UID: \"185ed8c1-2274-4cef-b764-caa9d400a138\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"
Jul 09 18:04:53.873174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.872994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f12fbd2f-6972-41b9-af69-b36fb3b29045-klusterlet-config\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.873174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/185ed8c1-2274-4cef-b764-caa9d400a138-nginx-conf\") pod \"networking-console-plugin-6cf856758c-6zkf5\" (UID: \"185ed8c1-2274-4cef-b764-caa9d400a138\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"
Jul 09 18:04:53.873174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:53.873174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edf89f9f-5e8c-41ca-899a-2e77089e36c5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f56d866d8-frswm\" (UID: \"edf89f9f-5e8c-41ca-899a-2e77089e36c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"
Jul 09 18:04:53.873174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f12fbd2f-6972-41b9-af69-b36fb3b29045-tmp\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.873174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz6hz\" (UniqueName: \"kubernetes.io/projected/75696b63-fe4e-4141-981f-81204168ea86-kube-api-access-bz6hz\") pod \"network-check-source-869dcfcdcd-h8zvk\" (UID: \"75696b63-fe4e-4141-981f-81204168ea86\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk"
Jul 09 18:04:53.873520 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlmw\" (UniqueName: \"kubernetes.io/projected/f12fbd2f-6972-41b9-af69-b36fb3b29045-kube-api-access-6dlmw\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.873520 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvj87\" (UniqueName: \"kubernetes.io/projected/edf89f9f-5e8c-41ca-899a-2e77089e36c5-kube-api-access-tvj87\") pod \"managed-serviceaccount-addon-agent-7f56d866d8-frswm\" (UID: \"edf89f9f-5e8c-41ca-899a-2e77089e36c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"
Jul 09 18:04:53.873823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f12fbd2f-6972-41b9-af69-b36fb3b29045-tmp\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.873919 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.873861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/185ed8c1-2274-4cef-b764-caa9d400a138-nginx-conf\") pod \"networking-console-plugin-6cf856758c-6zkf5\" (UID: \"185ed8c1-2274-4cef-b764-caa9d400a138\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"
Jul 09 18:04:53.875613 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.875592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f12fbd2f-6972-41b9-af69-b36fb3b29045-klusterlet-config\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.875709 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.875636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/185ed8c1-2274-4cef-b764-caa9d400a138-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-6zkf5\" (UID: \"185ed8c1-2274-4cef-b764-caa9d400a138\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"
Jul 09 18:04:53.875750 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.875723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/edf89f9f-5e8c-41ca-899a-2e77089e36c5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f56d866d8-frswm\" (UID: \"edf89f9f-5e8c-41ca-899a-2e77089e36c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"
Jul 09 18:04:53.876523 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.876509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6fjk"
Jul 09 18:04:53.879133 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.879114    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:04:53.889565 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.889540    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:04:53.894096 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.894072    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59bdd44495-fgpfm"]
Jul 09 18:04:53.896264 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.896237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-266zw\" (UniqueName: \"kubernetes.io/projected/c5affad4-9ffa-40cb-8a15-ff9c327859a0-kube-api-access-266zw\") pod \"network-check-target-ns4p4\" (UID: \"c5affad4-9ffa-40cb-8a15-ff9c327859a0\") " pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:53.904604 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.904558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlmw\" (UniqueName: \"kubernetes.io/projected/f12fbd2f-6972-41b9-af69-b36fb3b29045-kube-api-access-6dlmw\") pod \"klusterlet-addon-workmgr-79fc69b7-vt466\" (UID: \"f12fbd2f-6972-41b9-af69-b36fb3b29045\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:53.917422 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.917388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.963129 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.963091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59bdd44495-fgpfm"]
Jul 09 18:04:53.964071 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.963993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvj87\" (UniqueName: \"kubernetes.io/projected/edf89f9f-5e8c-41ca-899a-2e77089e36c5-kube-api-access-tvj87\") pod \"managed-serviceaccount-addon-agent-7f56d866d8-frswm\" (UID: \"edf89f9f-5e8c-41ca-899a-2e77089e36c5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"
Jul 09 18:04:53.974047 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-ca-trust-extracted\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.974184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-registry-certificates\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.974184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-trusted-ca\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.974184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7dh\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-kube-api-access-dt7dh\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.974184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz6hz\" (UniqueName: \"kubernetes.io/projected/75696b63-fe4e-4141-981f-81204168ea86-kube-api-access-bz6hz\") pod \"network-check-source-869dcfcdcd-h8zvk\" (UID: \"75696b63-fe4e-4141-981f-81204168ea86\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk"
Jul 09 18:04:53.974184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-image-registry-private-configuration\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.974184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-installation-pull-secrets\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.974472 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-bound-sa-token\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.974472 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.974336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-registry-tls\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:53.980040 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.980020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"
Jul 09 18:04:53.981528 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.981508    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jfsz2\""
Jul 09 18:04:53.987287 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.987228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sjz5q\""
Jul 09 18:04:53.989135 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.989118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:04:53.994305 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.994289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p55kt"
Jul 09 18:04:53.999178 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:53.999157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:04:54.003068 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.003047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz6hz\" (UniqueName: \"kubernetes.io/projected/75696b63-fe4e-4141-981f-81204168ea86-kube-api-access-bz6hz\") pod \"network-check-source-869dcfcdcd-h8zvk\" (UID: \"75696b63-fe4e-4141-981f-81204168ea86\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk"
Jul 09 18:04:54.027098 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.027071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"
Jul 09 18:04:54.052032 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.051874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6fjk"]
Jul 09 18:04:54.055646 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:54.055604    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50672b24_2d0d_4afc_b7de_ac97da06cb99.slice/crio-e5781eb384b70efc5bdaee6d52ab5c7fd2ffa3bcb08f04b87ce9ef9628571af1 WatchSource:0}: Error finding container e5781eb384b70efc5bdaee6d52ab5c7fd2ffa3bcb08f04b87ce9ef9628571af1: Status 404 returned error can't find the container with id e5781eb384b70efc5bdaee6d52ab5c7fd2ffa3bcb08f04b87ce9ef9628571af1
Jul 09 18:04:54.075791 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.075760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-ca-trust-extracted\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.075941 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.075809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-registry-certificates\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.075941 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.075845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-trusted-ca\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.075941 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.075883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7dh\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-kube-api-access-dt7dh\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.075941 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.075928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-image-registry-private-configuration\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.076154 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.075955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-installation-pull-secrets\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.076154 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.075985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-bound-sa-token\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.076154 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.076017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-registry-tls\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.077073 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.077007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-ca-trust-extracted\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.078478 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.077804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-trusted-ca\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.078478 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.078437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-registry-certificates\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.079343 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.078975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-registry-tls\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.079484 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.079461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-installation-pull-secrets\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.080753 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.080718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-image-registry-private-configuration\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.093789 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.093723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk"
Jul 09 18:04:54.101780 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.101712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7dh\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-kube-api-access-dt7dh\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.107567 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.107514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f-bound-sa-token\") pod \"image-registry-59bdd44495-fgpfm\" (UID: \"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f\") " pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.188449 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.186865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-6zkf5"]
Jul 09 18:04:54.198755 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:54.198698    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185ed8c1_2274_4cef_b764_caa9d400a138.slice/crio-81bd55ce371c862b4cf001dad8492fea9d06e057708974a33d313498b07ab0f7 WatchSource:0}: Error finding container 81bd55ce371c862b4cf001dad8492fea9d06e057708974a33d313498b07ab0f7: Status 404 returned error can't find the container with id 81bd55ce371c862b4cf001dad8492fea9d06e057708974a33d313498b07ab0f7
Jul 09 18:04:54.227747 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.227251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:54.235637 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.235553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ns4p4"]
Jul 09 18:04:54.240347 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:54.240308    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5affad4_9ffa_40cb_8a15_ff9c327859a0.slice/crio-7dc120e7a439f3f7d6abf14099a9d038a3b6ac3ffb1605aae710a2dc43816864 WatchSource:0}: Error finding container 7dc120e7a439f3f7d6abf14099a9d038a3b6ac3ffb1605aae710a2dc43816864: Status 404 returned error can't find the container with id 7dc120e7a439f3f7d6abf14099a9d038a3b6ac3ffb1605aae710a2dc43816864
Jul 09 18:04:54.254506 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.254456    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm"]
Jul 09 18:04:54.255429 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:54.255377    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedf89f9f_5e8c_41ca_899a_2e77089e36c5.slice/crio-70a65b7b45237395afa35a32d440a312ae4c53e741ef7f4cfb559eeddddea394 WatchSource:0}: Error finding container 70a65b7b45237395afa35a32d440a312ae4c53e741ef7f4cfb559eeddddea394: Status 404 returned error can't find the container with id 70a65b7b45237395afa35a32d440a312ae4c53e741ef7f4cfb559eeddddea394
Jul 09 18:04:54.288815 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.288774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm" event={"ID":"edf89f9f-5e8c-41ca-899a-2e77089e36c5","Type":"ContainerStarted","Data":"70a65b7b45237395afa35a32d440a312ae4c53e741ef7f4cfb559eeddddea394"}
Jul 09 18:04:54.290763 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.290660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5" event={"ID":"185ed8c1-2274-4cef-b764-caa9d400a138","Type":"ContainerStarted","Data":"81bd55ce371c862b4cf001dad8492fea9d06e057708974a33d313498b07ab0f7"}
Jul 09 18:04:54.292087 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.292057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ns4p4" event={"ID":"c5affad4-9ffa-40cb-8a15-ff9c327859a0","Type":"ContainerStarted","Data":"7dc120e7a439f3f7d6abf14099a9d038a3b6ac3ffb1605aae710a2dc43816864"}
Jul 09 18:04:54.293244 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.293213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6fjk" event={"ID":"50672b24-2d0d-4afc-b7de-ac97da06cb99","Type":"ContainerStarted","Data":"e5781eb384b70efc5bdaee6d52ab5c7fd2ffa3bcb08f04b87ce9ef9628571af1"}
Jul 09 18:04:54.341764 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.341737    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p55kt"]
Jul 09 18:04:54.342711 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:54.342681    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12fbd2f_6972_41b9_af69_b36fb3b29045.slice/crio-7aafabb9158647211117a1ebb39bd39bf72f13660868aa8405881d09eec5cc59 WatchSource:0}: Error finding container 7aafabb9158647211117a1ebb39bd39bf72f13660868aa8405881d09eec5cc59: Status 404 returned error can't find the container with id 7aafabb9158647211117a1ebb39bd39bf72f13660868aa8405881d09eec5cc59
Jul 09 18:04:54.342979 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.342959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"]
Jul 09 18:04:54.343467 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:54.343444    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9434adf9_7671_4b69_a142_438262daa264.slice/crio-f72b9bfd2c0d43b17db83888328f1458cde83dd4cc7a984e076bbd5784c37fec WatchSource:0}: Error finding container f72b9bfd2c0d43b17db83888328f1458cde83dd4cc7a984e076bbd5784c37fec: Status 404 returned error can't find the container with id f72b9bfd2c0d43b17db83888328f1458cde83dd4cc7a984e076bbd5784c37fec
Jul 09 18:04:54.368608 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.368502    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk"]
Jul 09 18:04:54.371756 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:54.371734    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75696b63_fe4e_4141_981f_81204168ea86.slice/crio-b8af2c0c211c8dcb8562f30bdd9b680f089563aadaa7077b4b35c9d048378d3d WatchSource:0}: Error finding container b8af2c0c211c8dcb8562f30bdd9b680f089563aadaa7077b4b35c9d048378d3d: Status 404 returned error can't find the container with id b8af2c0c211c8dcb8562f30bdd9b680f089563aadaa7077b4b35c9d048378d3d
Jul 09 18:04:54.411191 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:54.411131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59bdd44495-fgpfm"]
Jul 09 18:04:54.413743 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:04:54.413722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bd49d0_3b4d_462a_a7b6_3dc7cc9ce03f.slice/crio-1f37431846b6b83e54cdc1f746ad3379814e66eeab3b8c1e093f7f6d9ad12692 WatchSource:0}: Error finding container 1f37431846b6b83e54cdc1f746ad3379814e66eeab3b8c1e093f7f6d9ad12692: Status 404 returned error can't find the container with id 1f37431846b6b83e54cdc1f746ad3379814e66eeab3b8c1e093f7f6d9ad12692
Jul 09 18:04:55.303448 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:55.303352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk" event={"ID":"75696b63-fe4e-4141-981f-81204168ea86","Type":"ContainerStarted","Data":"b8af2c0c211c8dcb8562f30bdd9b680f089563aadaa7077b4b35c9d048378d3d"}
Jul 09 18:04:55.310030 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:55.309997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p55kt" event={"ID":"9434adf9-7671-4b69-a142-438262daa264","Type":"ContainerStarted","Data":"f72b9bfd2c0d43b17db83888328f1458cde83dd4cc7a984e076bbd5784c37fec"}
Jul 09 18:04:55.313475 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:55.313386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466" event={"ID":"f12fbd2f-6972-41b9-af69-b36fb3b29045","Type":"ContainerStarted","Data":"7aafabb9158647211117a1ebb39bd39bf72f13660868aa8405881d09eec5cc59"}
Jul 09 18:04:55.317550 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:55.316785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59bdd44495-fgpfm" event={"ID":"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f","Type":"ContainerStarted","Data":"8d988b25277541722e8d8d54891d14817de7ed841639d1fefb015074f22834d0"}
Jul 09 18:04:55.317550 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:55.316830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59bdd44495-fgpfm" event={"ID":"04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f","Type":"ContainerStarted","Data":"1f37431846b6b83e54cdc1f746ad3379814e66eeab3b8c1e093f7f6d9ad12692"}
Jul 09 18:04:55.317550 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:55.317501    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:04:55.347878 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:55.347340    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59bdd44495-fgpfm" podStartSLOduration=2.347320704 podStartE2EDuration="2.347320704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:55.345544109 +0000 UTC m=+67.795126534" watchObservedRunningTime="2026-07-09 18:04:55.347320704 +0000 UTC m=+67.796903109"
Jul 09 18:04:59.233316 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.233284    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"]
Jul 09 18:04:59.236420 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.236382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"
Jul 09 18:04:59.239141 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.239120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7n7hr\""
Jul 09 18:04:59.239965 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.239948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 09 18:04:59.251149 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.251116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"]
Jul 09 18:04:59.323424 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.323381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3fc53002-8732-41dd-9dd8-82f73a25d95a-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7kxt8\" (UID: \"3fc53002-8732-41dd-9dd8-82f73a25d95a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"
Jul 09 18:04:59.424390 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.424354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3fc53002-8732-41dd-9dd8-82f73a25d95a-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7kxt8\" (UID: \"3fc53002-8732-41dd-9dd8-82f73a25d95a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"
Jul 09 18:04:59.427005 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.426980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3fc53002-8732-41dd-9dd8-82f73a25d95a-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-7kxt8\" (UID: \"3fc53002-8732-41dd-9dd8-82f73a25d95a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"
Jul 09 18:04:59.547591 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:04:59.547521    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"
Jul 09 18:05:02.018269 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.018172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"]
Jul 09 18:05:02.352650 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.352554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5" event={"ID":"185ed8c1-2274-4cef-b764-caa9d400a138","Type":"ContainerStarted","Data":"cb2fed5b2ace360449f475f195f0a9563683b2674133e966a639970a6de04adf"}
Jul 09 18:05:02.354118 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.354093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ns4p4" event={"ID":"c5affad4-9ffa-40cb-8a15-ff9c327859a0","Type":"ContainerStarted","Data":"7d2eda08f73206a5215ac87556a9be5f1f699f73693d0ceaa6a9d9b4438b14ec"}
Jul 09 18:05:02.354251 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.354140    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:05:02.355556 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.355537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk" event={"ID":"75696b63-fe4e-4141-981f-81204168ea86","Type":"ContainerStarted","Data":"431f244ba7c46f14056bdb04a2cdd327be27793cd92f099ce9d714161845becb"}
Jul 09 18:05:02.357866 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.357841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p55kt" event={"ID":"9434adf9-7671-4b69-a142-438262daa264","Type":"ContainerStarted","Data":"08e12c879c95f51e3df7d09d55439e39b8bd3d1bf22cca1d6ed9ba0ee9f20e17"}
Jul 09 18:05:02.357979 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.357888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p55kt" event={"ID":"9434adf9-7671-4b69-a142-438262daa264","Type":"ContainerStarted","Data":"feab8e3ccff8f41087ef24267113713826b07b81d3301f0925b31188f66a347a"}
Jul 09 18:05:02.359237 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.359219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6fjk" event={"ID":"50672b24-2d0d-4afc-b7de-ac97da06cb99","Type":"ContainerStarted","Data":"1fc7d8478578a68ab43122697e3a22942939d0b0f020c51adc571485c7835b39"}
Jul 09 18:05:02.360593 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.360563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466" event={"ID":"f12fbd2f-6972-41b9-af69-b36fb3b29045","Type":"ContainerStarted","Data":"b66fde7c77152db84f3fbf4433f52abb0023ba9cb56cc52247bc02de4d156306"}
Jul 09 18:05:02.360708 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.360692    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:05:02.361741 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.361723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8" event={"ID":"3fc53002-8732-41dd-9dd8-82f73a25d95a","Type":"ContainerStarted","Data":"b6ffac4240f671de84af248d7d4bfacb9b67b0a853a7bb9dc2082d133ca912f6"}
Jul 09 18:05:02.362638 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.362620    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466"
Jul 09 18:05:02.370211 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.370174    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-6zkf5" podStartSLOduration=1.733951379 podStartE2EDuration="9.370161381s" totalImagesPullingTime="7.636210002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:02.36956255 +0000 UTC m=+74.819144975" watchObservedRunningTime="2026-07-09 18:05:02.370161381 +0000 UTC m=+74.819743804"
Jul 09 18:05:02.394587 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.394526    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79fc69b7-vt466" podStartSLOduration=1.864719462 podStartE2EDuration="9.394508299s" totalImagesPullingTime="7.529788837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:02.392788555 +0000 UTC m=+74.842370979" watchObservedRunningTime="2026-07-09 18:05:02.394508299 +0000 UTC m=+74.844090723"
Jul 09 18:05:02.416136 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.416083    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ns4p4" podStartSLOduration=66.803412835 podStartE2EDuration="1m14.416065685s" totalImagesPullingTime="7.61265285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:02.415767915 +0000 UTC m=+74.865350364" watchObservedRunningTime="2026-07-09 18:05:02.416065685 +0000 UTC m=+74.865648114"
Jul 09 18:05:02.439552 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.439491    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p55kt" podStartSLOduration=66.946726696 podStartE2EDuration="1m14.439478146s" totalImagesPullingTime="7.49275145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:02.437577467 +0000 UTC m=+74.887159908" watchObservedRunningTime="2026-07-09 18:05:02.439478146 +0000 UTC m=+74.889060570"
Jul 09 18:05:02.464445 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.464381    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-h8zvk" podStartSLOduration=1.987371435 podStartE2EDuration="9.464366232s" totalImagesPullingTime="7.476994797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:02.455640407 +0000 UTC m=+74.905222829" watchObservedRunningTime="2026-07-09 18:05:02.464366232 +0000 UTC m=+74.913948655"
Jul 09 18:05:02.482134 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:02.482086    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j6fjk" podStartSLOduration=66.675121787 podStartE2EDuration="1m14.482072985s" totalImagesPullingTime="7.806951198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:02.481278793 +0000 UTC m=+74.930861235" watchObservedRunningTime="2026-07-09 18:05:02.482072985 +0000 UTC m=+74.931655407"
Jul 09 18:05:03.724632 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:03.724601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:05:04.370972 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:04.370935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8" event={"ID":"3fc53002-8732-41dd-9dd8-82f73a25d95a","Type":"ContainerStarted","Data":"680dc1aa25fb62a5b43a64c66cfc80019081632d75d0cd1df6cc5d0cfc24ea30"}
Jul 09 18:05:04.371263 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:04.371232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"
Jul 09 18:05:04.375497 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:04.375481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8"
Jul 09 18:05:04.402818 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:04.402760    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-7kxt8" podStartSLOduration=3.943252802 podStartE2EDuration="5.402744274s" totalImagesPullingTime="1.459491472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:04.402702835 +0000 UTC m=+76.852285258" watchObservedRunningTime="2026-07-09 18:05:04.402744274 +0000 UTC m=+76.852326701"
Jul 09 18:05:11.241288 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.241154    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r5dxl"]
Jul 09 18:05:11.246124 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.246097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.249515 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.249491    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jnvlp\""
Jul 09 18:05:11.249850 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.249830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:05:11.250768 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.250610    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:05:11.250768 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.250624    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:05:11.250768 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.250697    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.250768 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.250717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.251023 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.250850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:05:11.416418 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-textfile\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.416599 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-tls\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.416599 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.416599 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-sys\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.416599 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flqwl\" (UniqueName: \"kubernetes.io/projected/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-kube-api-access-flqwl\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.416599 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-wtmp\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.416846 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.416846 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-root\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.416846 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.416696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-metrics-client-ca\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.517841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.517752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-sys\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.517841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.517791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flqwl\" (UniqueName: \"kubernetes.io/projected/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-kube-api-access-flqwl\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518037 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.517860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-sys\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518037 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.517928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-wtmp\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518037 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.517976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518181 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-root\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518181 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-metrics-client-ca\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518181 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-wtmp\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518181 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-textfile\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518181 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-root\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518181 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-tls\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518479 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518559 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-textfile\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518794 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.518931 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.518837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-metrics-client-ca\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.520880 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.520836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-tls\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.521038 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.521017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.526268 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.526247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flqwl\" (UniqueName: \"kubernetes.io/projected/6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8-kube-api-access-flqwl\") pod \"node-exporter-r5dxl\" (UID: \"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8\") " pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.557356 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:11.557321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r5dxl"
Jul 09 18:05:11.569007 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:05:11.568980    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fab0ae1_f8b1_4c8f_8eb0_ddab2ebb2da8.slice/crio-e45fed318574d29e59e82c04173f0cca6299e3ece486bdb595ea089747933552 WatchSource:0}: Error finding container e45fed318574d29e59e82c04173f0cca6299e3ece486bdb595ea089747933552: Status 404 returned error can't find the container with id e45fed318574d29e59e82c04173f0cca6299e3ece486bdb595ea089747933552
Jul 09 18:05:12.403561 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:12.403523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5dxl" event={"ID":"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8","Type":"ContainerStarted","Data":"e45fed318574d29e59e82c04173f0cca6299e3ece486bdb595ea089747933552"}
Jul 09 18:05:12.405352 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:12.405319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm" event={"ID":"edf89f9f-5e8c-41ca-899a-2e77089e36c5","Type":"ContainerStarted","Data":"7174fdb0a5342e74b43f0444f5d9f02100d1842d05b7b678aa7ba78b9c304c1d"}
Jul 09 18:05:12.424187 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:12.424134    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f56d866d8-frswm" podStartSLOduration=1.436758208 podStartE2EDuration="19.424113663s" totalImagesPullingTime="17.987355455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:12.422715541 +0000 UTC m=+84.872297990" watchObservedRunningTime="2026-07-09 18:05:12.424113663 +0000 UTC m=+84.873696086"
Jul 09 18:05:13.410498 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:13.410460    2563 generic.go:356] "Generic (PLEG): container finished" podID="6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8" containerID="6857f6590fcad5a79494d79fd0814dcd8d406d198cc54ff664e8807fa7944638" exitCode=0
Jul 09 18:05:13.410954 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:13.410549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5dxl" event={"ID":"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8","Type":"ContainerDied","Data":"6857f6590fcad5a79494d79fd0814dcd8d406d198cc54ff664e8807fa7944638"}
Jul 09 18:05:14.417356 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:14.417323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5dxl" event={"ID":"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8","Type":"ContainerStarted","Data":"89c9bb982881141b058be20ffc07993a12891dd00a5114a4a53885cafbbd1e31"}
Jul 09 18:05:14.417808 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:14.417385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5dxl" event={"ID":"6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8","Type":"ContainerStarted","Data":"89b7c3ed205267a9b8cb77e83559a51260902c4eb27fb9a3f5fee22f05001f23"}
Jul 09 18:05:14.440599 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:14.440553    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r5dxl" podStartSLOduration=2.195069509 podStartE2EDuration="3.440538038s" totalImagesPullingTime="1.18177638s" totalInitContainerRuntime="63.692149ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:14.439650134 +0000 UTC m=+86.889232557" watchObservedRunningTime="2026-07-09 18:05:14.440538038 +0000 UTC m=+86.890120461"
Jul 09 18:05:17.334340 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:17.334314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59bdd44495-fgpfm"
Jul 09 18:05:17.794738 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:17.794700    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-jnsfg"]
Jul 09 18:05:17.808237 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:17.808205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-jnsfg"
Jul 09 18:05:17.811076 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:17.811043    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hnbbr\""
Jul 09 18:05:17.811220 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:17.811114    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 18:05:17.811276 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:17.811217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 18:05:17.818005 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:17.817980    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-jnsfg"]
Jul 09 18:05:17.968447 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:17.968395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvbj\" (UniqueName: \"kubernetes.io/projected/7189db45-a477-45a9-85a3-a6af87915e9f-kube-api-access-lvvbj\") pod \"downloads-68c9cbb497-jnsfg\" (UID: \"7189db45-a477-45a9-85a3-a6af87915e9f\") " pod="openshift-console/downloads-68c9cbb497-jnsfg"
Jul 09 18:05:18.069158 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:18.069076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvbj\" (UniqueName: \"kubernetes.io/projected/7189db45-a477-45a9-85a3-a6af87915e9f-kube-api-access-lvvbj\") pod \"downloads-68c9cbb497-jnsfg\" (UID: \"7189db45-a477-45a9-85a3-a6af87915e9f\") " pod="openshift-console/downloads-68c9cbb497-jnsfg"
Jul 09 18:05:18.079554 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:18.079529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvbj\" (UniqueName: \"kubernetes.io/projected/7189db45-a477-45a9-85a3-a6af87915e9f-kube-api-access-lvvbj\") pod \"downloads-68c9cbb497-jnsfg\" (UID: \"7189db45-a477-45a9-85a3-a6af87915e9f\") " pod="openshift-console/downloads-68c9cbb497-jnsfg"
Jul 09 18:05:18.117827 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:18.117800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-jnsfg"
Jul 09 18:05:18.306218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:18.306181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-jnsfg"]
Jul 09 18:05:18.309864 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:05:18.309830    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7189db45_a477_45a9_85a3_a6af87915e9f.slice/crio-fca42ab902eae89f11b90ec77d205ecf510c790ae5cdc41f8500f32f5c480a6f WatchSource:0}: Error finding container fca42ab902eae89f11b90ec77d205ecf510c790ae5cdc41f8500f32f5c480a6f: Status 404 returned error can't find the container with id fca42ab902eae89f11b90ec77d205ecf510c790ae5cdc41f8500f32f5c480a6f
Jul 09 18:05:18.431571 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:18.431533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-jnsfg" event={"ID":"7189db45-a477-45a9-85a3-a6af87915e9f","Type":"ContainerStarted","Data":"fca42ab902eae89f11b90ec77d205ecf510c790ae5cdc41f8500f32f5c480a6f"}
Jul 09 18:05:18.739469 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:18.739354    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5795676c7-jvqjh" podUID="173eee0a-fa09-40c3-9348-396facbed712" containerName="registry" containerID="cri-o://8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3" gracePeriod=30
Jul 09 18:05:19.006431 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.006343    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:05:19.177081 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-bound-sa-token\") pod \"173eee0a-fa09-40c3-9348-396facbed712\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") "
Jul 09 18:05:19.177250 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-image-registry-private-configuration\") pod \"173eee0a-fa09-40c3-9348-396facbed712\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") "
Jul 09 18:05:19.177250 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177142    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-trusted-ca\") pod \"173eee0a-fa09-40c3-9348-396facbed712\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") "
Jul 09 18:05:19.177250 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177188    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-registry-certificates\") pod \"173eee0a-fa09-40c3-9348-396facbed712\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") "
Jul 09 18:05:19.177250 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-installation-pull-secrets\") pod \"173eee0a-fa09-40c3-9348-396facbed712\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") "
Jul 09 18:05:19.177442 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/173eee0a-fa09-40c3-9348-396facbed712-ca-trust-extracted\") pod \"173eee0a-fa09-40c3-9348-396facbed712\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") "
Jul 09 18:05:19.177442 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177297    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") pod \"173eee0a-fa09-40c3-9348-396facbed712\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") "
Jul 09 18:05:19.177442 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfwms\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-kube-api-access-qfwms\") pod \"173eee0a-fa09-40c3-9348-396facbed712\" (UID: \"173eee0a-fa09-40c3-9348-396facbed712\") "
Jul 09 18:05:19.177813 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177647    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "173eee0a-fa09-40c3-9348-396facbed712" (UID: "173eee0a-fa09-40c3-9348-396facbed712"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:19.177813 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.177659    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "173eee0a-fa09-40c3-9348-396facbed712" (UID: "173eee0a-fa09-40c3-9348-396facbed712"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:05:19.179882 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.179803    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "173eee0a-fa09-40c3-9348-396facbed712" (UID: "173eee0a-fa09-40c3-9348-396facbed712"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:19.179882 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.179845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "173eee0a-fa09-40c3-9348-396facbed712" (UID: "173eee0a-fa09-40c3-9348-396facbed712"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:19.180059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.180004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "173eee0a-fa09-40c3-9348-396facbed712" (UID: "173eee0a-fa09-40c3-9348-396facbed712"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:19.180100 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.180056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "173eee0a-fa09-40c3-9348-396facbed712" (UID: "173eee0a-fa09-40c3-9348-396facbed712"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:19.180135 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.180104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-kube-api-access-qfwms" (OuterVolumeSpecName: "kube-api-access-qfwms") pod "173eee0a-fa09-40c3-9348-396facbed712" (UID: "173eee0a-fa09-40c3-9348-396facbed712"). InnerVolumeSpecName "kube-api-access-qfwms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:19.186368 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.186344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173eee0a-fa09-40c3-9348-396facbed712-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "173eee0a-fa09-40c3-9348-396facbed712" (UID: "173eee0a-fa09-40c3-9348-396facbed712"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:19.278841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.278758    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-bound-sa-token\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:05:19.278841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.278784    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-image-registry-private-configuration\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:05:19.278841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.278794    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-trusted-ca\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:05:19.278841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.278802    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/173eee0a-fa09-40c3-9348-396facbed712-registry-certificates\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:05:19.278841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.278811    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/173eee0a-fa09-40c3-9348-396facbed712-installation-pull-secrets\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:05:19.278841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.278826    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/173eee0a-fa09-40c3-9348-396facbed712-ca-trust-extracted\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:05:19.278841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.278835    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-registry-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:05:19.278841 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.278843    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfwms\" (UniqueName: \"kubernetes.io/projected/173eee0a-fa09-40c3-9348-396facbed712-kube-api-access-qfwms\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:05:19.437063 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.437032    2563 generic.go:356] "Generic (PLEG): container finished" podID="173eee0a-fa09-40c3-9348-396facbed712" containerID="8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3" exitCode=0
Jul 09 18:05:19.437463 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.437081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5795676c7-jvqjh" event={"ID":"173eee0a-fa09-40c3-9348-396facbed712","Type":"ContainerDied","Data":"8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3"}
Jul 09 18:05:19.437463 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.437087    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5795676c7-jvqjh"
Jul 09 18:05:19.437463 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.437127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5795676c7-jvqjh" event={"ID":"173eee0a-fa09-40c3-9348-396facbed712","Type":"ContainerDied","Data":"5174cfd819df4f2c4275e59d9387a359420cd4454609795854a9a8a6eaeb8625"}
Jul 09 18:05:19.437463 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.437149    2563 scope.go:117] "RemoveContainer" containerID="8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3"
Jul 09 18:05:19.445383 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.445366    2563 scope.go:117] "RemoveContainer" containerID="8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3"
Jul 09 18:05:19.445685 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:05:19.445663    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3\": container with ID starting with 8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3 not found: ID does not exist" containerID="8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3"
Jul 09 18:05:19.445740 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.445698    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3"} err="failed to get container status \"8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3\": rpc error: code = NotFound desc = could not find container \"8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3\": container with ID starting with 8b2134142dd5f713ae2805e0812dc068d0908ac93d9f609500492734dcf92db3 not found: ID does not exist"
Jul 09 18:05:19.470168 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.470142    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5795676c7-jvqjh"]
Jul 09 18:05:19.473122 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.473104    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5795676c7-jvqjh"]
Jul 09 18:05:19.973054 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:19.973018    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="173eee0a-fa09-40c3-9348-396facbed712" path="/var/lib/kubelet/pods/173eee0a-fa09-40c3-9348-396facbed712/volumes"
Jul 09 18:05:29.695263 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.695228    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d8876bc6f-rzj9g"]
Jul 09 18:05:29.695831 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.695577    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173eee0a-fa09-40c3-9348-396facbed712" containerName="registry"
Jul 09 18:05:29.695831 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.695594    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="173eee0a-fa09-40c3-9348-396facbed712" containerName="registry"
Jul 09 18:05:29.695831 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.695668    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="173eee0a-fa09-40c3-9348-396facbed712" containerName="registry"
Jul 09 18:05:29.698845 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.698817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.701527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.701502    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:05:29.702507 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.702485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:05:29.702614 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.702485    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:05:29.702614 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.702520    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6b4cn\""
Jul 09 18:05:29.702614 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.702511    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:05:29.702754 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.702655    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:05:29.713482 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.713457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8876bc6f-rzj9g"]
Jul 09 18:05:29.864269 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.864232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6rq\" (UniqueName: \"kubernetes.io/projected/4eb13a65-6475-4abb-8d9f-f00dc227972d-kube-api-access-vv6rq\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.864492 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.864287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-oauth-config\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.864492 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.864341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-config\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.864492 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.864371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-service-ca\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.864492 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.864441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-oauth-serving-cert\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.864664 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.864507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-serving-cert\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.965594 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.965506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-serving-cert\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.965594 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.965570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6rq\" (UniqueName: \"kubernetes.io/projected/4eb13a65-6475-4abb-8d9f-f00dc227972d-kube-api-access-vv6rq\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.965835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.965612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-oauth-config\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.965835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.965658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-config\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.965835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.965687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-service-ca\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.965835 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.965713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-oauth-serving-cert\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.966553 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.966455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-service-ca\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.966553 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.966504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-config\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.966553 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.966504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-oauth-serving-cert\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.968333 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.968310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-oauth-config\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.968604 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.968579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-serving-cert\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:29.975690 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:29.975668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6rq\" (UniqueName: \"kubernetes.io/projected/4eb13a65-6475-4abb-8d9f-f00dc227972d-kube-api-access-vv6rq\") pod \"console-6d8876bc6f-rzj9g\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") " pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:30.010576 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:30.010543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:33.368012 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:33.367976    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ns4p4"
Jul 09 18:05:35.992234 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:35.992203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8876bc6f-rzj9g"]
Jul 09 18:05:35.994977 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:05:35.994941    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb13a65_6475_4abb_8d9f_f00dc227972d.slice/crio-c765b0ecf2cae036e25688decf9504f8e8bbcb98b5d2c5cae0d75eaccf26fac2 WatchSource:0}: Error finding container c765b0ecf2cae036e25688decf9504f8e8bbcb98b5d2c5cae0d75eaccf26fac2: Status 404 returned error can't find the container with id c765b0ecf2cae036e25688decf9504f8e8bbcb98b5d2c5cae0d75eaccf26fac2
Jul 09 18:05:36.511168 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:36.511129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-jnsfg" event={"ID":"7189db45-a477-45a9-85a3-a6af87915e9f","Type":"ContainerStarted","Data":"513bdae237fcabbbbf7a2f71055efa436712abf2cf231d2a58bf678f457648c7"}
Jul 09 18:05:36.511367 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:36.511221    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-jnsfg"
Jul 09 18:05:36.512476 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:36.512444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8876bc6f-rzj9g" event={"ID":"4eb13a65-6475-4abb-8d9f-f00dc227972d","Type":"ContainerStarted","Data":"c765b0ecf2cae036e25688decf9504f8e8bbcb98b5d2c5cae0d75eaccf26fac2"}
Jul 09 18:05:36.512724 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:36.512703    2563 patch_prober.go:28] interesting pod/downloads-68c9cbb497-jnsfg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.19:8080/\": dial tcp 10.133.0.19:8080: connect: connection refused" start-of-body=
Jul 09 18:05:36.512796 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:36.512755    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-jnsfg" podUID="7189db45-a477-45a9-85a3-a6af87915e9f" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.19:8080/\": dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:05:36.530520 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:36.530479    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-jnsfg" podStartSLOduration=1.551184871 podStartE2EDuration="19.530467116s" totalImagesPullingTime="17.979282245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:36.528659668 +0000 UTC m=+108.978242117" watchObservedRunningTime="2026-07-09 18:05:36.530467116 +0000 UTC m=+108.980049543"
Jul 09 18:05:37.531948 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:37.531916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-jnsfg"
Jul 09 18:05:39.864803 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.864767    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-599f77d7b4-j8zwb"]
Jul 09 18:05:39.916113 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.916049    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599f77d7b4-j8zwb"]
Jul 09 18:05:39.916291 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.916190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:39.931966 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.931937    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 18:05:39.951047 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.951012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-config\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:39.951197 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.951161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-trusted-ca-bundle\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:39.951266 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.951211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-serving-cert\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:39.951362 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.951337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-service-ca\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:39.951443 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.951384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-oauth-serving-cert\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:39.951516 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.951478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghdzg\" (UniqueName: \"kubernetes.io/projected/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-kube-api-access-ghdzg\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:39.951579 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:39.951564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-oauth-config\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.052423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.052369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-trusted-ca-bundle\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.052423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.052435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-serving-cert\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.052844 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.052475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-service-ca\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.052844 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.052504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-oauth-serving-cert\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.052844 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.052536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghdzg\" (UniqueName: \"kubernetes.io/projected/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-kube-api-access-ghdzg\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.052844 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.052585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-oauth-config\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.052844 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.052641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-config\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.053495 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.053463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-trusted-ca-bundle\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.053905 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.053743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-service-ca\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.054012 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.053934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-oauth-serving-cert\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.054230 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.054189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-config\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.056475 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.056445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-oauth-config\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.056935 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.056897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-serving-cert\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.069487 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.069462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghdzg\" (UniqueName: \"kubernetes.io/projected/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-kube-api-access-ghdzg\") pod \"console-599f77d7b4-j8zwb\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") " pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.231002 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.230965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:40.426821 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.426751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599f77d7b4-j8zwb"]
Jul 09 18:05:40.481109 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:05:40.481032    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ce5ad3_5a45_4384_9bcb_3c3de2a1b42d.slice/crio-819bd23249b2104104b8e21910815e0ebec0a6d1a028dc2d7a54d3af5a281ffe WatchSource:0}: Error finding container 819bd23249b2104104b8e21910815e0ebec0a6d1a028dc2d7a54d3af5a281ffe: Status 404 returned error can't find the container with id 819bd23249b2104104b8e21910815e0ebec0a6d1a028dc2d7a54d3af5a281ffe
Jul 09 18:05:40.538419 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:40.538371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599f77d7b4-j8zwb" event={"ID":"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d","Type":"ContainerStarted","Data":"819bd23249b2104104b8e21910815e0ebec0a6d1a028dc2d7a54d3af5a281ffe"}
Jul 09 18:05:41.544646 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:41.544606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599f77d7b4-j8zwb" event={"ID":"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d","Type":"ContainerStarted","Data":"0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81"}
Jul 09 18:05:41.546629 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:41.546599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8876bc6f-rzj9g" event={"ID":"4eb13a65-6475-4abb-8d9f-f00dc227972d","Type":"ContainerStarted","Data":"fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84"}
Jul 09 18:05:41.569240 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:41.569192    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-599f77d7b4-j8zwb" podStartSLOduration=2.569172781 podStartE2EDuration="2.569172781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:41.568705977 +0000 UTC m=+114.018288404" watchObservedRunningTime="2026-07-09 18:05:41.569172781 +0000 UTC m=+114.018755205"
Jul 09 18:05:50.010832 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.010792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:50.011228 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.010845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:50.015690 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.015670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:50.036541 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.036491    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d8876bc6f-rzj9g" podStartSLOduration=16.525999624 podStartE2EDuration="21.036475888s" totalImagesPullingTime="4.510476264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:41.595565787 +0000 UTC m=+114.045148211" watchObservedRunningTime="2026-07-09 18:05:50.036475888 +0000 UTC m=+122.486058311"
Jul 09 18:05:50.231847 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.231807    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:50.232015 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.231878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:50.236499 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.236475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:50.586716 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.586686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:05:50.586894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.586749    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:05:50.668544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:50.668515    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d8876bc6f-rzj9g"]
Jul 09 18:05:52.591646 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:52.591614    2563 generic.go:356] "Generic (PLEG): container finished" podID="01a20cc4-485d-40ff-847e-c0034eeaa8c5" containerID="98850aa5b6e9e925ecabb11f01fe9080a269e010cdba4cd105ae11a13bf48837" exitCode=0
Jul 09 18:05:52.592207 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:52.591688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z" event={"ID":"01a20cc4-485d-40ff-847e-c0034eeaa8c5","Type":"ContainerDied","Data":"98850aa5b6e9e925ecabb11f01fe9080a269e010cdba4cd105ae11a13bf48837"}
Jul 09 18:05:52.592207 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:52.592051    2563 scope.go:117] "RemoveContainer" containerID="98850aa5b6e9e925ecabb11f01fe9080a269e010cdba4cd105ae11a13bf48837"
Jul 09 18:05:53.597057 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:53.597021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ftb9z" event={"ID":"01a20cc4-485d-40ff-847e-c0034eeaa8c5","Type":"ContainerStarted","Data":"c5ccb0eb5618ecdbbc28752bbe46dbef99c1771e390173393f58da25a4094ac5"}
Jul 09 18:05:53.793420 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:53.793377    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5dxl_6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8/init-textfile/0.log"
Jul 09 18:05:53.995265 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:53.995232    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5dxl_6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8/node-exporter/0.log"
Jul 09 18:05:54.194111 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:54.194082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5dxl_6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8/kube-rbac-proxy/0.log"
Jul 09 18:05:56.796746 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:56.796717    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-7kxt8_3fc53002-8732-41dd-9dd8-82f73a25d95a/prometheus-operator-admission-webhook/0.log"
Jul 09 18:05:58.793681 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:58.793653    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-6zkf5_185ed8c1-2274-4cef-b764-caa9d400a138/networking-console-plugin/0.log"
Jul 09 18:05:59.393918 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:59.393889    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-599f77d7b4-j8zwb_d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d/console/0.log"
Jul 09 18:05:59.594536 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:59.594506    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d8876bc6f-rzj9g_4eb13a65-6475-4abb-8d9f-f00dc227972d/console/0.log"
Jul 09 18:05:59.797429 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:05:59.797386    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-jnsfg_7189db45-a477-45a9-85a3-a6af87915e9f/download-server/0.log"
Jul 09 18:06:17.615369 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:17.615304    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d8876bc6f-rzj9g" podUID="4eb13a65-6475-4abb-8d9f-f00dc227972d" containerName="console" containerID="cri-o://fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84" gracePeriod=15
Jul 09 18:06:17.894797 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:17.894723    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:06:18.064238 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-oauth-serving-cert\") pod \"4eb13a65-6475-4abb-8d9f-f00dc227972d\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") "
Jul 09 18:06:18.064394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064258    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-oauth-config\") pod \"4eb13a65-6475-4abb-8d9f-f00dc227972d\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") "
Jul 09 18:06:18.064394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-config\") pod \"4eb13a65-6475-4abb-8d9f-f00dc227972d\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") "
Jul 09 18:06:18.064394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-service-ca\") pod \"4eb13a65-6475-4abb-8d9f-f00dc227972d\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") "
Jul 09 18:06:18.064394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064350    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv6rq\" (UniqueName: \"kubernetes.io/projected/4eb13a65-6475-4abb-8d9f-f00dc227972d-kube-api-access-vv6rq\") pod \"4eb13a65-6475-4abb-8d9f-f00dc227972d\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") "
Jul 09 18:06:18.064603 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-serving-cert\") pod \"4eb13a65-6475-4abb-8d9f-f00dc227972d\" (UID: \"4eb13a65-6475-4abb-8d9f-f00dc227972d\") "
Jul 09 18:06:18.064700 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4eb13a65-6475-4abb-8d9f-f00dc227972d" (UID: "4eb13a65-6475-4abb-8d9f-f00dc227972d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:18.064781 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064761    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-config" (OuterVolumeSpecName: "console-config") pod "4eb13a65-6475-4abb-8d9f-f00dc227972d" (UID: "4eb13a65-6475-4abb-8d9f-f00dc227972d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:18.064845 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.064774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-service-ca" (OuterVolumeSpecName: "service-ca") pod "4eb13a65-6475-4abb-8d9f-f00dc227972d" (UID: "4eb13a65-6475-4abb-8d9f-f00dc227972d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:18.066780 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.066743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb13a65-6475-4abb-8d9f-f00dc227972d-kube-api-access-vv6rq" (OuterVolumeSpecName: "kube-api-access-vv6rq") pod "4eb13a65-6475-4abb-8d9f-f00dc227972d" (UID: "4eb13a65-6475-4abb-8d9f-f00dc227972d"). InnerVolumeSpecName "kube-api-access-vv6rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:18.066856 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.066793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4eb13a65-6475-4abb-8d9f-f00dc227972d" (UID: "4eb13a65-6475-4abb-8d9f-f00dc227972d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:18.066856 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.066802    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4eb13a65-6475-4abb-8d9f-f00dc227972d" (UID: "4eb13a65-6475-4abb-8d9f-f00dc227972d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:18.165921 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.165843    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-oauth-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.165921 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.165871    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.165921 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.165881    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-service-ca\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.165921 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.165889    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv6rq\" (UniqueName: \"kubernetes.io/projected/4eb13a65-6475-4abb-8d9f-f00dc227972d-kube-api-access-vv6rq\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.165921 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.165898    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb13a65-6475-4abb-8d9f-f00dc227972d-console-serving-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.165921 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.165906    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb13a65-6475-4abb-8d9f-f00dc227972d-oauth-serving-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:06:18.692549 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.692518    2563 generic.go:356] "Generic (PLEG): container finished" podID="4eb13a65-6475-4abb-8d9f-f00dc227972d" containerID="fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84" exitCode=0
Jul 09 18:06:18.692983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.692569    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8876bc6f-rzj9g"
Jul 09 18:06:18.692983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.692596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8876bc6f-rzj9g" event={"ID":"4eb13a65-6475-4abb-8d9f-f00dc227972d","Type":"ContainerDied","Data":"fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84"}
Jul 09 18:06:18.692983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.692641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8876bc6f-rzj9g" event={"ID":"4eb13a65-6475-4abb-8d9f-f00dc227972d","Type":"ContainerDied","Data":"c765b0ecf2cae036e25688decf9504f8e8bbcb98b5d2c5cae0d75eaccf26fac2"}
Jul 09 18:06:18.692983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.692662    2563 scope.go:117] "RemoveContainer" containerID="fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84"
Jul 09 18:06:18.701314 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.701294    2563 scope.go:117] "RemoveContainer" containerID="fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84"
Jul 09 18:06:18.701591 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:06:18.701572    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84\": container with ID starting with fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84 not found: ID does not exist" containerID="fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84"
Jul 09 18:06:18.701639 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.701600    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84"} err="failed to get container status \"fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84\": rpc error: code = NotFound desc = could not find container \"fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84\": container with ID starting with fca06a3f4f89cbb6bc2cdf7eaa7f3c3183e71646a6ad7edbc31910e0155f0f84 not found: ID does not exist"
Jul 09 18:06:18.714543 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.714521    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d8876bc6f-rzj9g"]
Jul 09 18:06:18.718520 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:18.718503    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d8876bc6f-rzj9g"]
Jul 09 18:06:19.971768 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:06:19.971738    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb13a65-6475-4abb-8d9f-f00dc227972d" path="/var/lib/kubelet/pods/4eb13a65-6475-4abb-8d9f-f00dc227972d/volumes"
Jul 09 18:07:17.972767 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:17.972741    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-685c49f889-dqv2k"]
Jul 09 18:07:17.973218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:17.972970    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb13a65-6475-4abb-8d9f-f00dc227972d" containerName="console"
Jul 09 18:07:17.973218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:17.972979    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb13a65-6475-4abb-8d9f-f00dc227972d" containerName="console"
Jul 09 18:07:17.973218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:17.973029    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb13a65-6475-4abb-8d9f-f00dc227972d" containerName="console"
Jul 09 18:07:17.975763 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:17.975746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:17.992680 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:17.992659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-685c49f889-dqv2k"]
Jul 09 18:07:18.097153 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.097119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8h7\" (UniqueName: \"kubernetes.io/projected/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-kube-api-access-tr8h7\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.097153 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.097162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-config\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.097357 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.097193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-oauth-serving-cert\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.097357 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.097328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-oauth-config\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.097534 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.097513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-trusted-ca-bundle\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.097669 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.097606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-serving-cert\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.097669 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.097645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-service-ca\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.198489 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.198460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-trusted-ca-bundle\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.198652 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.198506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-serving-cert\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.198652 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.198531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-service-ca\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.198652 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.198557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8h7\" (UniqueName: \"kubernetes.io/projected/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-kube-api-access-tr8h7\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.198652 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.198591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-config\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.198652 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.198617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-oauth-serving-cert\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.198894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.198744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-oauth-config\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.199267 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.199243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-config\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.199392 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.199320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-service-ca\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.199392 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.199352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-oauth-serving-cert\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.199392 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.199380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-trusted-ca-bundle\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.200858 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.200835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-serving-cert\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.201071 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.201052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-oauth-config\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.207097 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.207076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8h7\" (UniqueName: \"kubernetes.io/projected/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-kube-api-access-tr8h7\") pod \"console-685c49f889-dqv2k\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") " pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.284653 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.284596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:18.411615 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.411457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-685c49f889-dqv2k"]
Jul 09 18:07:18.413684 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:07:18.413658    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59287cd_345b_4733_ad9d_4d7c4ab5bd3b.slice/crio-238571e575560fdfa6a43ffeacff5154fba111833fc25cd69df78ca52ca9cffb WatchSource:0}: Error finding container 238571e575560fdfa6a43ffeacff5154fba111833fc25cd69df78ca52ca9cffb: Status 404 returned error can't find the container with id 238571e575560fdfa6a43ffeacff5154fba111833fc25cd69df78ca52ca9cffb
Jul 09 18:07:18.908789 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.908757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685c49f889-dqv2k" event={"ID":"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b","Type":"ContainerStarted","Data":"1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd"}
Jul 09 18:07:18.908962 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.908804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685c49f889-dqv2k" event={"ID":"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b","Type":"ContainerStarted","Data":"238571e575560fdfa6a43ffeacff5154fba111833fc25cd69df78ca52ca9cffb"}
Jul 09 18:07:18.926617 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:18.926568    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-685c49f889-dqv2k" podStartSLOduration=1.926552931 podStartE2EDuration="1.926552931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:07:18.925739572 +0000 UTC m=+211.375321995" watchObservedRunningTime="2026-07-09 18:07:18.926552931 +0000 UTC m=+211.376135778"
Jul 09 18:07:28.285501 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:28.285472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:28.285876 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:28.285554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:28.290209 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:28.290191    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:28.947155 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:28.947127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:07:28.996212 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:28.996184    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599f77d7b4-j8zwb"]
Jul 09 18:07:40.563867 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.563832    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"]
Jul 09 18:07:40.567159 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.567142    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.570128 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.570112    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 18:07:40.570128 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.570121    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 18:07:40.570269 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.570112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dzr5p\""
Jul 09 18:07:40.576669 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.576651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"]
Jul 09 18:07:40.686784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.686753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.686939 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.686799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.686939 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.686828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggw6v\" (UniqueName: \"kubernetes.io/projected/86c83ce2-992e-495e-9f24-cae5aeee1d06-kube-api-access-ggw6v\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.788008 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.787980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.788114 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.788026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.788114 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.788059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggw6v\" (UniqueName: \"kubernetes.io/projected/86c83ce2-992e-495e-9f24-cae5aeee1d06-kube-api-access-ggw6v\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.788371 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.788349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.788465 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.788375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.797664 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.797633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggw6v\" (UniqueName: \"kubernetes.io/projected/86c83ce2-992e-495e-9f24-cae5aeee1d06-kube-api-access-ggw6v\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:40.876527 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:40.876437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:07:41.009950 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:41.009815    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"]
Jul 09 18:07:41.012611 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:07:41.012580    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c83ce2_992e_495e_9f24_cae5aeee1d06.slice/crio-098b0193ef8fd63d34784fe979556893d96f824bbd79339f1796c72f27851a36 WatchSource:0}: Error finding container 098b0193ef8fd63d34784fe979556893d96f824bbd79339f1796c72f27851a36: Status 404 returned error can't find the container with id 098b0193ef8fd63d34784fe979556893d96f824bbd79339f1796c72f27851a36
Jul 09 18:07:41.991900 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:41.991866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw" event={"ID":"86c83ce2-992e-495e-9f24-cae5aeee1d06","Type":"ContainerStarted","Data":"098b0193ef8fd63d34784fe979556893d96f824bbd79339f1796c72f27851a36"}
Jul 09 18:07:49.017611 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:49.017572    2563 generic.go:356] "Generic (PLEG): container finished" podID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerID="76144366b330f32623cf4091d95724c10d2980b5a51f298b5050a28c61ddb9d9" exitCode=0
Jul 09 18:07:49.017898 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:49.017650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw" event={"ID":"86c83ce2-992e-495e-9f24-cae5aeee1d06","Type":"ContainerDied","Data":"76144366b330f32623cf4091d95724c10d2980b5a51f298b5050a28c61ddb9d9"}
Jul 09 18:07:54.016790 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.016746    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-599f77d7b4-j8zwb" podUID="d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" containerName="console" containerID="cri-o://0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81" gracePeriod=15
Jul 09 18:07:54.251978 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.251952    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:07:54.286627 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.286572    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-trusted-ca-bundle\") pod \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") "
Jul 09 18:07:54.286627 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.286611    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-oauth-config\") pod \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") "
Jul 09 18:07:54.286771 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.286718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-serving-cert\") pod \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") "
Jul 09 18:07:54.286771 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.286764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-service-ca\") pod \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") "
Jul 09 18:07:54.286854 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.286792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-config\") pod \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") "
Jul 09 18:07:54.286854 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.286818    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghdzg\" (UniqueName: \"kubernetes.io/projected/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-kube-api-access-ghdzg\") pod \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") "
Jul 09 18:07:54.286854 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.286849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-oauth-serving-cert\") pod \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\" (UID: \"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d\") "
Jul 09 18:07:54.287153 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.287129    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-config" (OuterVolumeSpecName: "console-config") pod "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" (UID: "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:54.287320 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.287187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-service-ca" (OuterVolumeSpecName: "service-ca") pod "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" (UID: "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:54.287382 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.287363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" (UID: "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:54.287473 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.287454    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" (UID: "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:54.288744 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.288716    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" (UID: "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:07:54.289233 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.289208    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" (UID: "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:07:54.289233 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.289221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-kube-api-access-ghdzg" (OuterVolumeSpecName: "kube-api-access-ghdzg") pod "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" (UID: "d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d"). InnerVolumeSpecName "kube-api-access-ghdzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:07:54.387615 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.387594    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-trusted-ca-bundle\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:07:54.387615 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.387614    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-oauth-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:07:54.387737 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.387623    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-serving-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:07:54.387737 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.387631    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-service-ca\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:07:54.387737 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.387640    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-console-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:07:54.387737 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.387648    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghdzg\" (UniqueName: \"kubernetes.io/projected/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-kube-api-access-ghdzg\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:07:54.387737 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:54.387657    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d-oauth-serving-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:07:55.041081 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.041043    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" containerID="0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81" exitCode=0
Jul 09 18:07:55.041552 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.041088    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599f77d7b4-j8zwb"
Jul 09 18:07:55.041552 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.041131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599f77d7b4-j8zwb" event={"ID":"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d","Type":"ContainerDied","Data":"0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81"}
Jul 09 18:07:55.041552 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.041189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599f77d7b4-j8zwb" event={"ID":"d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d","Type":"ContainerDied","Data":"819bd23249b2104104b8e21910815e0ebec0a6d1a028dc2d7a54d3af5a281ffe"}
Jul 09 18:07:55.041552 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.041223    2563 scope.go:117] "RemoveContainer" containerID="0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81"
Jul 09 18:07:55.049337 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.049322    2563 scope.go:117] "RemoveContainer" containerID="0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81"
Jul 09 18:07:55.049602 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:07:55.049584    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81\": container with ID starting with 0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81 not found: ID does not exist" containerID="0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81"
Jul 09 18:07:55.049672 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.049607    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81"} err="failed to get container status \"0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81\": rpc error: code = NotFound desc = could not find container \"0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81\": container with ID starting with 0101c050482216c411e579fdc6cf11df788d571edee1516a4ff9fd8da32ebf81 not found: ID does not exist"
Jul 09 18:07:55.061146 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.061124    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599f77d7b4-j8zwb"]
Jul 09 18:07:55.064585 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.064565    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-599f77d7b4-j8zwb"]
Jul 09 18:07:55.971237 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:07:55.971202    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" path="/var/lib/kubelet/pods/d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d/volumes"
Jul 09 18:08:13.106345 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:13.106310    2563 generic.go:356] "Generic (PLEG): container finished" podID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerID="7c7f82283492423d9d87c4831d25c51ea6a1dc005ef3938bcdd3ee54a053c71f" exitCode=0
Jul 09 18:08:13.106854 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:13.106364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw" event={"ID":"86c83ce2-992e-495e-9f24-cae5aeee1d06","Type":"ContainerDied","Data":"7c7f82283492423d9d87c4831d25c51ea6a1dc005ef3938bcdd3ee54a053c71f"}
Jul 09 18:08:22.142650 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:22.142617    2563 generic.go:356] "Generic (PLEG): container finished" podID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerID="155fca290aa6d7b1760908c14f4fe333f6a8edf5fd9df0293087af1a9aa8af79" exitCode=0
Jul 09 18:08:22.143327 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:22.142702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw" event={"ID":"86c83ce2-992e-495e-9f24-cae5aeee1d06","Type":"ContainerDied","Data":"155fca290aa6d7b1760908c14f4fe333f6a8edf5fd9df0293087af1a9aa8af79"}
Jul 09 18:08:23.264252 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.264227    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:08:23.404119 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.404043    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggw6v\" (UniqueName: \"kubernetes.io/projected/86c83ce2-992e-495e-9f24-cae5aeee1d06-kube-api-access-ggw6v\") pod \"86c83ce2-992e-495e-9f24-cae5aeee1d06\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") "
Jul 09 18:08:23.404119 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.404078    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-bundle\") pod \"86c83ce2-992e-495e-9f24-cae5aeee1d06\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") "
Jul 09 18:08:23.404119 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.404104    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-util\") pod \"86c83ce2-992e-495e-9f24-cae5aeee1d06\" (UID: \"86c83ce2-992e-495e-9f24-cae5aeee1d06\") "
Jul 09 18:08:23.404784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.404761    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-bundle" (OuterVolumeSpecName: "bundle") pod "86c83ce2-992e-495e-9f24-cae5aeee1d06" (UID: "86c83ce2-992e-495e-9f24-cae5aeee1d06"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:23.406173 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.406152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c83ce2-992e-495e-9f24-cae5aeee1d06-kube-api-access-ggw6v" (OuterVolumeSpecName: "kube-api-access-ggw6v") pod "86c83ce2-992e-495e-9f24-cae5aeee1d06" (UID: "86c83ce2-992e-495e-9f24-cae5aeee1d06"). InnerVolumeSpecName "kube-api-access-ggw6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:08:23.408095 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.408070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-util" (OuterVolumeSpecName: "util") pod "86c83ce2-992e-495e-9f24-cae5aeee1d06" (UID: "86c83ce2-992e-495e-9f24-cae5aeee1d06"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:23.505536 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.505508    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggw6v\" (UniqueName: \"kubernetes.io/projected/86c83ce2-992e-495e-9f24-cae5aeee1d06-kube-api-access-ggw6v\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:08:23.505536 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.505531    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-bundle\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:08:23.505536 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:23.505540    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86c83ce2-992e-495e-9f24-cae5aeee1d06-util\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:08:24.151033 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:24.151002    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw"
Jul 09 18:08:24.151221 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:24.151000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb5898dsgw" event={"ID":"86c83ce2-992e-495e-9f24-cae5aeee1d06","Type":"ContainerDied","Data":"098b0193ef8fd63d34784fe979556893d96f824bbd79339f1796c72f27851a36"}
Jul 09 18:08:24.151221 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:24.151117    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="098b0193ef8fd63d34784fe979556893d96f824bbd79339f1796c72f27851a36"
Jul 09 18:08:34.469782 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.469743    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-v9g8s"]
Jul 09 18:08:34.470289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470135    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerName="extract"
Jul 09 18:08:34.470289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470152    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerName="extract"
Jul 09 18:08:34.470289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470169    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" containerName="console"
Jul 09 18:08:34.470289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470178    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" containerName="console"
Jul 09 18:08:34.470289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerName="util"
Jul 09 18:08:34.470289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470197    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerName="util"
Jul 09 18:08:34.470289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470220    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerName="pull"
Jul 09 18:08:34.470289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470227    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerName="pull"
Jul 09 18:08:34.470685 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470295    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ce5ad3-5a45-4384-9bcb-3c3de2a1b42d" containerName="console"
Jul 09 18:08:34.470685 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.470307    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c83ce2-992e-495e-9f24-cae5aeee1d06" containerName="extract"
Jul 09 18:08:34.473523 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.473502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:34.479545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.476854    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 09 18:08:34.479545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.477182    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 09 18:08:34.479545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.477672    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 09 18:08:34.479545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.477909    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 09 18:08:34.479545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.478130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 09 18:08:34.479545 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.478359    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zcwwj\""
Jul 09 18:08:34.489081 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.489061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-v9g8s"]
Jul 09 18:08:34.594610 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.594574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qxd4\" (UniqueName: \"kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-kube-api-access-7qxd4\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:34.594808 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.594626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-certificates\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:34.594808 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.594744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9ea64895-3284-42aa-8875-45fbc064adcb-cabundle0\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:34.695421 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.695383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9ea64895-3284-42aa-8875-45fbc064adcb-cabundle0\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:34.695610 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.695455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qxd4\" (UniqueName: \"kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-kube-api-access-7qxd4\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:34.695610 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.695504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-certificates\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:34.695610 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:08:34.695602    2563 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 09 18:08:34.695767 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:08:34.695619    2563 secret.go:281] references non-existent secret key: ca.crt
Jul 09 18:08:34.695767 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:08:34.695629    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 09 18:08:34.695767 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:08:34.695643    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-v9g8s: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 09 18:08:34.695767 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:08:34.695706    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-certificates podName:9ea64895-3284-42aa-8875-45fbc064adcb nodeName:}" failed. No retries permitted until 2026-07-09 18:08:35.195686001 +0000 UTC m=+287.645268410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-certificates") pod "keda-operator-68bff59c-v9g8s" (UID: "9ea64895-3284-42aa-8875-45fbc064adcb") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 09 18:08:34.696030 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.696009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9ea64895-3284-42aa-8875-45fbc064adcb-cabundle0\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:34.704605 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:34.704582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qxd4\" (UniqueName: \"kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-kube-api-access-7qxd4\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:35.081157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.081121    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"]
Jul 09 18:08:35.084289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.084272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.086845 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.086829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 09 18:08:35.096360 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.096336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"]
Jul 09 18:08:35.199713 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.199683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-certificates\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:35.199890 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.199764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c7b610b2-898c-401d-915e-29c99a0192e9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.199890 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.199800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7b610b2-898c-401d-915e-29c99a0192e9-certificates\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.199890 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.199823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvnh\" (UniqueName: \"kubernetes.io/projected/c7b610b2-898c-401d-915e-29c99a0192e9-kube-api-access-qtvnh\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.202020 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.201992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ea64895-3284-42aa-8875-45fbc064adcb-certificates\") pod \"keda-operator-68bff59c-v9g8s\" (UID: \"9ea64895-3284-42aa-8875-45fbc064adcb\") " pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:35.301057 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.301018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7b610b2-898c-401d-915e-29c99a0192e9-certificates\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.301057 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.301057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvnh\" (UniqueName: \"kubernetes.io/projected/c7b610b2-898c-401d-915e-29c99a0192e9-kube-api-access-qtvnh\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.301303 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.301240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c7b610b2-898c-401d-915e-29c99a0192e9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.301645 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.301623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c7b610b2-898c-401d-915e-29c99a0192e9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.303509 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.303487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7b610b2-898c-401d-915e-29c99a0192e9-certificates\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.310028 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.310003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvnh\" (UniqueName: \"kubernetes.io/projected/c7b610b2-898c-401d-915e-29c99a0192e9-kube-api-access-qtvnh\") pod \"keda-metrics-apiserver-559f75f947-hw47r\" (UID: \"c7b610b2-898c-401d-915e-29c99a0192e9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.388334 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.388253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:35.393283 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.393261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:35.550090 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.550057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-v9g8s"]
Jul 09 18:08:35.553289 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:08:35.553256    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea64895_3284_42aa_8875_45fbc064adcb.slice/crio-9ba35da32e541dcf0feef2b1445b45d39bc2d03c53f8e502b6599d0c6388af35 WatchSource:0}: Error finding container 9ba35da32e541dcf0feef2b1445b45d39bc2d03c53f8e502b6599d0c6388af35: Status 404 returned error can't find the container with id 9ba35da32e541dcf0feef2b1445b45d39bc2d03c53f8e502b6599d0c6388af35
Jul 09 18:08:35.559882 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:35.559859    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"]
Jul 09 18:08:35.561279 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:08:35.561258    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b610b2_898c_401d_915e_29c99a0192e9.slice/crio-0a126960e6f20397552ea147a13f66c8b7b334476132028e3ad59079e6c1252d WatchSource:0}: Error finding container 0a126960e6f20397552ea147a13f66c8b7b334476132028e3ad59079e6c1252d: Status 404 returned error can't find the container with id 0a126960e6f20397552ea147a13f66c8b7b334476132028e3ad59079e6c1252d
Jul 09 18:08:36.194715 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:36.194667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r" event={"ID":"c7b610b2-898c-401d-915e-29c99a0192e9","Type":"ContainerStarted","Data":"0a126960e6f20397552ea147a13f66c8b7b334476132028e3ad59079e6c1252d"}
Jul 09 18:08:36.195967 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:36.195940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-v9g8s" event={"ID":"9ea64895-3284-42aa-8875-45fbc064adcb","Type":"ContainerStarted","Data":"9ba35da32e541dcf0feef2b1445b45d39bc2d03c53f8e502b6599d0c6388af35"}
Jul 09 18:08:38.208989 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:38.208954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r" event={"ID":"c7b610b2-898c-401d-915e-29c99a0192e9","Type":"ContainerStarted","Data":"4420b714d995ebdd75afda214df6dbaebd6450e348bd60aefc00eaa68feed490"}
Jul 09 18:08:38.209441 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:38.209046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:08:38.227236 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:38.227178    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r" podStartSLOduration=0.877442156 podStartE2EDuration="3.227163343s" totalImagesPullingTime="2.349721187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:38.22649341 +0000 UTC m=+290.676075850" watchObservedRunningTime="2026-07-09 18:08:38.227163343 +0000 UTC m=+290.676745764"
Jul 09 18:08:39.214466 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:39.214433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-v9g8s" event={"ID":"9ea64895-3284-42aa-8875-45fbc064adcb","Type":"ContainerStarted","Data":"71fbdba9a0da1c52b64e2b441625c50c6cd380cf6141008c3b3b5e97f6d170fd"}
Jul 09 18:08:39.214870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:39.214503    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:08:39.233459 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:39.233417    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-v9g8s" podStartSLOduration=1.745376831 podStartE2EDuration="5.233385703s" totalImagesPullingTime="3.488008872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:39.231704944 +0000 UTC m=+291.681287379" watchObservedRunningTime="2026-07-09 18:08:39.233385703 +0000 UTC m=+291.682968124"
Jul 09 18:08:47.855342 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:47.855316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:08:47.856625 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:47.856607    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:08:47.858725 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:47.858709    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:08:49.218483 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:08:49.218450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hw47r"
Jul 09 18:09:00.220580 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:00.220544    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-v9g8s"
Jul 09 18:09:43.237917 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.237831    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-c2nkw"]
Jul 09 18:09:43.240982 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.240961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:43.243623 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.243604    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:09:43.243703 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.243620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tj2ht\""
Jul 09 18:09:43.244337 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.244322    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 18:09:43.244394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.244378    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:09:43.250508 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.250489    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-c2nkw"]
Jul 09 18:09:43.327115 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.327075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a787447b-6580-4fbd-8ffd-a179e13cb70d-data\") pod \"seaweedfs-64568bcd49-c2nkw\" (UID: \"a787447b-6580-4fbd-8ffd-a179e13cb70d\") " pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:43.327296 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.327140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf5nr\" (UniqueName: \"kubernetes.io/projected/a787447b-6580-4fbd-8ffd-a179e13cb70d-kube-api-access-jf5nr\") pod \"seaweedfs-64568bcd49-c2nkw\" (UID: \"a787447b-6580-4fbd-8ffd-a179e13cb70d\") " pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:43.427633 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.427592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a787447b-6580-4fbd-8ffd-a179e13cb70d-data\") pod \"seaweedfs-64568bcd49-c2nkw\" (UID: \"a787447b-6580-4fbd-8ffd-a179e13cb70d\") " pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:43.427799 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.427731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf5nr\" (UniqueName: \"kubernetes.io/projected/a787447b-6580-4fbd-8ffd-a179e13cb70d-kube-api-access-jf5nr\") pod \"seaweedfs-64568bcd49-c2nkw\" (UID: \"a787447b-6580-4fbd-8ffd-a179e13cb70d\") " pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:43.427984 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.427964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a787447b-6580-4fbd-8ffd-a179e13cb70d-data\") pod \"seaweedfs-64568bcd49-c2nkw\" (UID: \"a787447b-6580-4fbd-8ffd-a179e13cb70d\") " pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:43.436646 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.436616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf5nr\" (UniqueName: \"kubernetes.io/projected/a787447b-6580-4fbd-8ffd-a179e13cb70d-kube-api-access-jf5nr\") pod \"seaweedfs-64568bcd49-c2nkw\" (UID: \"a787447b-6580-4fbd-8ffd-a179e13cb70d\") " pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:43.551332 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.551240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:43.682637 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.682603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-c2nkw"]
Jul 09 18:09:43.686037 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:09:43.686011    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda787447b_6580_4fbd_8ffd_a179e13cb70d.slice/crio-3cb58b45b9fbfb66ab7399df1246f9ee44e2a770719133e4d90e094240ca7f35 WatchSource:0}: Error finding container 3cb58b45b9fbfb66ab7399df1246f9ee44e2a770719133e4d90e094240ca7f35: Status 404 returned error can't find the container with id 3cb58b45b9fbfb66ab7399df1246f9ee44e2a770719133e4d90e094240ca7f35
Jul 09 18:09:43.687847 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:43.687828    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:09:44.448795 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:44.448751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-c2nkw" event={"ID":"a787447b-6580-4fbd-8ffd-a179e13cb70d","Type":"ContainerStarted","Data":"3cb58b45b9fbfb66ab7399df1246f9ee44e2a770719133e4d90e094240ca7f35"}
Jul 09 18:09:48.464419 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:48.464381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-c2nkw" event={"ID":"a787447b-6580-4fbd-8ffd-a179e13cb70d","Type":"ContainerStarted","Data":"751ddc803220e8fa4ff319d031987dc086377aaba7166513d717c6534cd67e53"}
Jul 09 18:09:48.464887 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:48.464467    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:09:48.482491 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:48.482443    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-c2nkw" podStartSLOduration=1.673650562 podStartE2EDuration="5.48242853s" totalImagesPullingTime="3.808777968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:48.481093733 +0000 UTC m=+360.930676166" watchObservedRunningTime="2026-07-09 18:09:48.48242853 +0000 UTC m=+360.932010982"
Jul 09 18:09:54.470474 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:09:54.470440    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-c2nkw"
Jul 09 18:10:43.441321 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.441281    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8675cfc849-56w68"]
Jul 09 18:10:43.444537 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.444515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.460631 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.460599    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8675cfc849-56w68"]
Jul 09 18:10:43.581699 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.581655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-trusted-ca-bundle\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.581880 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.581710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-oauth-config\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.581880 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.581761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-service-ca\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.581880 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.581801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-oauth-serving-cert\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.581880 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.581845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-config\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.582001 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.581905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b2wp\" (UniqueName: \"kubernetes.io/projected/7e95a149-6c15-4bb1-97f1-e7f44faef832-kube-api-access-8b2wp\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.582001 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.581937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-serving-cert\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.683170 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.683129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b2wp\" (UniqueName: \"kubernetes.io/projected/7e95a149-6c15-4bb1-97f1-e7f44faef832-kube-api-access-8b2wp\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.683170 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.683169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-serving-cert\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.683459 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.683212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-trusted-ca-bundle\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.683459 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.683258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-oauth-config\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.683459 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.683370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-service-ca\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.683459 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.683442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-oauth-serving-cert\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.683665 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.683484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-config\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.684148 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.684120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-config\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.684249 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.684229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-service-ca\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.684289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.684262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-oauth-serving-cert\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.684323 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.684300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e95a149-6c15-4bb1-97f1-e7f44faef832-trusted-ca-bundle\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.686242 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.686222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-serving-cert\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.686394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.686373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e95a149-6c15-4bb1-97f1-e7f44faef832-console-oauth-config\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.691550 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.691498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b2wp\" (UniqueName: \"kubernetes.io/projected/7e95a149-6c15-4bb1-97f1-e7f44faef832-kube-api-access-8b2wp\") pod \"console-8675cfc849-56w68\" (UID: \"7e95a149-6c15-4bb1-97f1-e7f44faef832\") " pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.754612 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.754570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:43.890602 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:43.890574    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8675cfc849-56w68"]
Jul 09 18:10:43.893101 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:10:43.893072    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e95a149_6c15_4bb1_97f1_e7f44faef832.slice/crio-76a6e617a09c21ae12d7ffad65cdeed33be124feaa080bfe51660411bf6e563b WatchSource:0}: Error finding container 76a6e617a09c21ae12d7ffad65cdeed33be124feaa080bfe51660411bf6e563b: Status 404 returned error can't find the container with id 76a6e617a09c21ae12d7ffad65cdeed33be124feaa080bfe51660411bf6e563b
Jul 09 18:10:44.672049 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:44.672008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8675cfc849-56w68" event={"ID":"7e95a149-6c15-4bb1-97f1-e7f44faef832","Type":"ContainerStarted","Data":"73f2845726f48f0de1863f2e09bd64f026ea586189679841f2305da6cc2704aa"}
Jul 09 18:10:44.672520 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:44.672078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8675cfc849-56w68" event={"ID":"7e95a149-6c15-4bb1-97f1-e7f44faef832","Type":"ContainerStarted","Data":"76a6e617a09c21ae12d7ffad65cdeed33be124feaa080bfe51660411bf6e563b"}
Jul 09 18:10:44.694105 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:44.694050    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8675cfc849-56w68" podStartSLOduration=1.6940315689999998 podStartE2EDuration="1.694031569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:44.691861371 +0000 UTC m=+417.141443816" watchObservedRunningTime="2026-07-09 18:10:44.694031569 +0000 UTC m=+417.143613992"
Jul 09 18:10:53.755422 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:53.755368    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:53.755968 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:53.755442    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:53.760418 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:53.760381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:54.105124 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.105040    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-c2jqn"]
Jul 09 18:10:54.108105 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.108083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:54.110509 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.110474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 18:10:54.110709 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.110694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-dlj8h\""
Jul 09 18:10:54.126322 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.126297    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-c2jqn"]
Jul 09 18:10:54.176535 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.176489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cqc\" (UniqueName: \"kubernetes.io/projected/ab481449-e45a-41fa-8edd-025edab5691e-kube-api-access-s5cqc\") pod \"model-serving-api-fd65d7d6-c2jqn\" (UID: \"ab481449-e45a-41fa-8edd-025edab5691e\") " pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:54.176716 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.176643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab481449-e45a-41fa-8edd-025edab5691e-tls-certs\") pod \"model-serving-api-fd65d7d6-c2jqn\" (UID: \"ab481449-e45a-41fa-8edd-025edab5691e\") " pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:54.277778 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.277747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cqc\" (UniqueName: \"kubernetes.io/projected/ab481449-e45a-41fa-8edd-025edab5691e-kube-api-access-s5cqc\") pod \"model-serving-api-fd65d7d6-c2jqn\" (UID: \"ab481449-e45a-41fa-8edd-025edab5691e\") " pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:54.277949 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.277792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab481449-e45a-41fa-8edd-025edab5691e-tls-certs\") pod \"model-serving-api-fd65d7d6-c2jqn\" (UID: \"ab481449-e45a-41fa-8edd-025edab5691e\") " pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:54.280495 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.280472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab481449-e45a-41fa-8edd-025edab5691e-tls-certs\") pod \"model-serving-api-fd65d7d6-c2jqn\" (UID: \"ab481449-e45a-41fa-8edd-025edab5691e\") " pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:54.288761 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.288740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cqc\" (UniqueName: \"kubernetes.io/projected/ab481449-e45a-41fa-8edd-025edab5691e-kube-api-access-s5cqc\") pod \"model-serving-api-fd65d7d6-c2jqn\" (UID: \"ab481449-e45a-41fa-8edd-025edab5691e\") " pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:54.418022 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.417995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:54.547968 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.547890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-c2jqn"]
Jul 09 18:10:54.550659 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:10:54.550627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab481449_e45a_41fa_8edd_025edab5691e.slice/crio-119267e081f1491d8b1622deb6b4d1828717eae83c33beb9e8e6405e7d724dc0 WatchSource:0}: Error finding container 119267e081f1491d8b1622deb6b4d1828717eae83c33beb9e8e6405e7d724dc0: Status 404 returned error can't find the container with id 119267e081f1491d8b1622deb6b4d1828717eae83c33beb9e8e6405e7d724dc0
Jul 09 18:10:54.719965 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.719872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-c2jqn" event={"ID":"ab481449-e45a-41fa-8edd-025edab5691e","Type":"ContainerStarted","Data":"119267e081f1491d8b1622deb6b4d1828717eae83c33beb9e8e6405e7d724dc0"}
Jul 09 18:10:54.723805 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.723787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8675cfc849-56w68"
Jul 09 18:10:54.782508 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:54.782470    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-685c49f889-dqv2k"]
Jul 09 18:10:57.734753 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:57.734722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-c2jqn" event={"ID":"ab481449-e45a-41fa-8edd-025edab5691e","Type":"ContainerStarted","Data":"f4155b894e25acd6ba9335b6385f422308b37b9ff9e8ae99901e8eedfa2a5fb4"}
Jul 09 18:10:57.735110 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:57.734863    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:10:57.755261 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:10:57.755217    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-c2jqn" podStartSLOduration=1.440752373 podStartE2EDuration="3.755203286s" totalImagesPullingTime="2.314450913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:57.753832487 +0000 UTC m=+430.203414931" watchObservedRunningTime="2026-07-09 18:10:57.755203286 +0000 UTC m=+430.204785712"
Jul 09 18:11:08.742608 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:08.742579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-c2jqn"
Jul 09 18:11:19.803935 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:19.803840    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-685c49f889-dqv2k" podUID="a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" containerName="console" containerID="cri-o://1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd" gracePeriod=15
Jul 09 18:11:20.047954 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.047931    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:11:20.079951 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.079870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-config\") pod \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") "
Jul 09 18:11:20.079951 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.079907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-service-ca\") pod \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") "
Jul 09 18:11:20.079951 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.079949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-oauth-config\") pod \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") "
Jul 09 18:11:20.080208 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080003    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr8h7\" (UniqueName: \"kubernetes.io/projected/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-kube-api-access-tr8h7\") pod \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") "
Jul 09 18:11:20.080208 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-trusted-ca-bundle\") pod \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") "
Jul 09 18:11:20.080208 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-serving-cert\") pod \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") "
Jul 09 18:11:20.080420 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080375    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-service-ca" (OuterVolumeSpecName: "service-ca") pod "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" (UID: "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:20.080475 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080440    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-config" (OuterVolumeSpecName: "console-config") pod "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" (UID: "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:20.080529 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080477    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-oauth-serving-cert\") pod \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\" (UID: \"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b\") "
Jul 09 18:11:20.081100 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080735    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.081100 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080757    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-service-ca\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.081100 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.080870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" (UID: "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:20.081100 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.081057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" (UID: "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:20.082952 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.082928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" (UID: "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:20.083058 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.082948    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-kube-api-access-tr8h7" (OuterVolumeSpecName: "kube-api-access-tr8h7") pod "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" (UID: "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b"). InnerVolumeSpecName "kube-api-access-tr8h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:11:20.083058 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.083016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" (UID: "a59287cd-345b-4733-ad9d-4d7c4ab5bd3b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:20.181617 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.181584    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-oauth-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.181617 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.181609    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr8h7\" (UniqueName: \"kubernetes.io/projected/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-kube-api-access-tr8h7\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.181617 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.181618    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-trusted-ca-bundle\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.181617 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.181627    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-console-serving-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.181863 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.181636    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b-oauth-serving-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:11:20.825460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.825399    2563 generic.go:356] "Generic (PLEG): container finished" podID="a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" containerID="1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd" exitCode=0
Jul 09 18:11:20.825460 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.825461    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685c49f889-dqv2k"
Jul 09 18:11:20.825919 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.825499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685c49f889-dqv2k" event={"ID":"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b","Type":"ContainerDied","Data":"1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd"}
Jul 09 18:11:20.825919 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.825550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685c49f889-dqv2k" event={"ID":"a59287cd-345b-4733-ad9d-4d7c4ab5bd3b","Type":"ContainerDied","Data":"238571e575560fdfa6a43ffeacff5154fba111833fc25cd69df78ca52ca9cffb"}
Jul 09 18:11:20.825919 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.825571    2563 scope.go:117] "RemoveContainer" containerID="1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd"
Jul 09 18:11:20.834278 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.834262    2563 scope.go:117] "RemoveContainer" containerID="1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd"
Jul 09 18:11:20.834606 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:11:20.834588    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd\": container with ID starting with 1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd not found: ID does not exist" containerID="1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd"
Jul 09 18:11:20.834683 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.834619    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd"} err="failed to get container status \"1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd\": rpc error: code = NotFound desc = could not find container \"1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd\": container with ID starting with 1d64e5aabaa522fe47120ed3f31263dcba161c2a6e97b23d9ef42061c2ebeddd not found: ID does not exist"
Jul 09 18:11:20.850921 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.850897    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-685c49f889-dqv2k"]
Jul 09 18:11:20.854808 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:20.854785    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-685c49f889-dqv2k"]
Jul 09 18:11:21.971344 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:11:21.971302    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" path="/var/lib/kubelet/pods/a59287cd-345b-4733-ad9d-4d7c4ab5bd3b/volumes"
Jul 09 18:12:44.265120 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.265043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"]
Jul 09 18:12:44.265572 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.265344    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" containerName="console"
Jul 09 18:12:44.265572 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.265356    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" containerName="console"
Jul 09 18:12:44.265572 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.265416    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a59287cd-345b-4733-ad9d-4d7c4ab5bd3b" containerName="console"
Jul 09 18:12:44.268506 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.268489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.271391 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.271370    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:12:44.271636 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.271618    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:12:44.272239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.272225    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-c3a93-predictor-serving-cert\""
Jul 09 18:12:44.273790 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.273774    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8d4cq\""
Jul 09 18:12:44.273883 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.273846    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-c3a93-kube-rbac-proxy-sar-config\""
Jul 09 18:12:44.274096 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.274082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:12:44.274299 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.274284    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:12:44.292147 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.292125    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"]
Jul 09 18:12:44.315601 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.315572    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37832215-3d18-429f-829d-757ba6479270-proxy-tls\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.315773 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.315613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.315773 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.315645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37832215-3d18-429f-829d-757ba6479270-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.315773 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.315682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.315773 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.315717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdkk\" (UniqueName: \"kubernetes.io/projected/37832215-3d18-429f-829d-757ba6479270-kube-api-access-jpdkk\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.416218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.416185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdkk\" (UniqueName: \"kubernetes.io/projected/37832215-3d18-429f-829d-757ba6479270-kube-api-access-jpdkk\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.416384 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.416248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37832215-3d18-429f-829d-757ba6479270-proxy-tls\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.416384 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.416283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.416384 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.416308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37832215-3d18-429f-829d-757ba6479270-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.416384 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.416345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.416824 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.416792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37832215-3d18-429f-829d-757ba6479270-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.417048 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.417029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.417117 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.417084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.418776 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.418756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37832215-3d18-429f-829d-757ba6479270-proxy-tls\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.427856 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.427832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdkk\" (UniqueName: \"kubernetes.io/projected/37832215-3d18-429f-829d-757ba6479270-kube-api-access-jpdkk\") pod \"isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.578288 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.578204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:12:44.719935 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:44.719903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"]
Jul 09 18:12:44.723042 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:12:44.723015    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37832215_3d18_429f_829d_757ba6479270.slice/crio-24fd6beacc317ac36457eac774f4c3daf209f746bff544e8b788742b4d59a209 WatchSource:0}: Error finding container 24fd6beacc317ac36457eac774f4c3daf209f746bff544e8b788742b4d59a209: Status 404 returned error can't find the container with id 24fd6beacc317ac36457eac774f4c3daf209f746bff544e8b788742b4d59a209
Jul 09 18:12:45.141572 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:45.141534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerStarted","Data":"24fd6beacc317ac36457eac774f4c3daf209f746bff544e8b788742b4d59a209"}
Jul 09 18:12:48.919828 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:48.919805    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:12:49.159239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:49.159202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerStarted","Data":"a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42"}
Jul 09 18:12:53.176104 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:53.176063    2563 generic.go:356] "Generic (PLEG): container finished" podID="37832215-3d18-429f-829d-757ba6479270" containerID="a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42" exitCode=0
Jul 09 18:12:53.176518 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:12:53.176137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerDied","Data":"a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42"}
Jul 09 18:13:11.261039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:11.260961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerStarted","Data":"9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4"}
Jul 09 18:13:14.275861 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:14.275825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerStarted","Data":"33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794"}
Jul 09 18:13:17.290762 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:17.290727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerStarted","Data":"db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0"}
Jul 09 18:13:17.291157 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:17.290958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:13:17.324332 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:17.324283    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podStartSLOduration=1.710306388 podStartE2EDuration="33.324269111s" totalImagesPullingTime="27.685288332s" totalInitContainerRuntime="3.928674391s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:44 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:17.323770762 +0000 UTC m=+569.773353185" watchObservedRunningTime="2026-07-09 18:13:17.324269111 +0000 UTC m=+569.773851592"
Jul 09 18:13:18.294977 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:18.294947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:13:18.295351 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:18.295046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:13:18.296384 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:18.296345    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:13:18.297066 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:18.297041    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:13:18.299869 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:18.299851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:13:19.298759 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:19.298719    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:13:19.299141 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:19.299001    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:13:20.302152 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:20.302111    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:13:20.302628 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:20.302444    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:13:30.302788 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:30.302750    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:13:30.303284 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:30.303258    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:13:40.302456 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:40.302397    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:13:40.302928 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:40.302757    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:13:47.881051 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:47.881020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:13:47.881602 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:47.881462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:13:50.302684 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:50.302640    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:13:50.305292 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:13:50.303075    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:00.302843 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:00.302797    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:14:00.303270 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:00.303245    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:10.302252 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:10.302162    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:14:10.302706 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:10.302683    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:20.302153 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:20.302107    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:14:20.302646 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:20.302515    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:30.302904 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:30.302868    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:14:30.303267 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:30.303011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:14:39.451725 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.451691    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"]
Jul 09 18:14:39.452235 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.452153    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" containerID="cri-o://9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4" gracePeriod=30
Jul 09 18:14:39.452235 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.452178    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy" containerID="cri-o://33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794" gracePeriod=30
Jul 09 18:14:39.452363 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.452169    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" containerID="cri-o://db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0" gracePeriod=30
Jul 09 18:14:39.564161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.564119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"]
Jul 09 18:14:39.567909 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.567888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.570313 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.570281    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a2b25-kube-rbac-proxy-sar-config\""
Jul 09 18:14:39.570454 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.570321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a2b25-predictor-serving-cert\""
Jul 09 18:14:39.579221 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.579156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"]
Jul 09 18:14:39.615349 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.615318    2563 generic.go:356] "Generic (PLEG): container finished" podID="37832215-3d18-429f-829d-757ba6479270" containerID="33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794" exitCode=2
Jul 09 18:14:39.615530 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.615358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerDied","Data":"33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794"}
Jul 09 18:14:39.650856 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.650827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vbc\" (UniqueName: \"kubernetes.io/projected/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-api-access-r8vbc\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.650982 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.650881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.650982 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.650911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beb5ce5-17f9-4332-a248-33241e5ab02a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.650982 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.650948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.651106 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.651043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.752429 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.752326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vbc\" (UniqueName: \"kubernetes.io/projected/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-api-access-r8vbc\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.752429 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.752394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.752625 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.752442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beb5ce5-17f9-4332-a248-33241e5ab02a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.752625 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.752481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.752625 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.752541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.752625 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:14:39.752541    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-serving-cert: secret "isvc-sklearn-graph-raw-a2b25-predictor-serving-cert" not found
Jul 09 18:14:39.752625 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:14:39.752618    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls podName:8beb5ce5-17f9-4332-a248-33241e5ab02a nodeName:}" failed. No retries permitted until 2026-07-09 18:14:40.252595456 +0000 UTC m=+652.702177858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls") pod "isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" (UID: "8beb5ce5-17f9-4332-a248-33241e5ab02a") : secret "isvc-sklearn-graph-raw-a2b25-predictor-serving-cert" not found
Jul 09 18:14:39.752952 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.752929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beb5ce5-17f9-4332-a248-33241e5ab02a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.753316 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.753299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.753360 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.753342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:39.761788 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:39.761756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vbc\" (UniqueName: \"kubernetes.io/projected/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-api-access-r8vbc\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:40.256177 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:40.256148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:40.258515 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:40.258486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls\") pod \"isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:40.302916 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:40.302877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:14:40.303182 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:40.303158    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:40.482211 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:40.482164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:40.614034 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:40.614008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"]
Jul 09 18:14:40.616203 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:14:40.616173    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8beb5ce5_17f9_4332_a248_33241e5ab02a.slice/crio-14ff6e5a97d948cd2aef199c332872ab9609a71552baa03334a1641b6721c76d WatchSource:0}: Error finding container 14ff6e5a97d948cd2aef199c332872ab9609a71552baa03334a1641b6721c76d: Status 404 returned error can't find the container with id 14ff6e5a97d948cd2aef199c332872ab9609a71552baa03334a1641b6721c76d
Jul 09 18:14:41.625238 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:41.625198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" event={"ID":"8beb5ce5-17f9-4332-a248-33241e5ab02a","Type":"ContainerStarted","Data":"0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37"}
Jul 09 18:14:41.625627 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:41.625258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" event={"ID":"8beb5ce5-17f9-4332-a248-33241e5ab02a","Type":"ContainerStarted","Data":"14ff6e5a97d948cd2aef199c332872ab9609a71552baa03334a1641b6721c76d"}
Jul 09 18:14:43.295485 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:43.295446    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 09 18:14:44.639827 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:44.639800    2563 generic.go:356] "Generic (PLEG): container finished" podID="37832215-3d18-429f-829d-757ba6479270" containerID="9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4" exitCode=0
Jul 09 18:14:44.640220 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:44.639874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerDied","Data":"9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4"}
Jul 09 18:14:44.641376 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:44.641354    2563 generic.go:356] "Generic (PLEG): container finished" podID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerID="0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37" exitCode=0
Jul 09 18:14:44.641481 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:44.641432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" event={"ID":"8beb5ce5-17f9-4332-a248-33241e5ab02a","Type":"ContainerDied","Data":"0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37"}
Jul 09 18:14:45.649003 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:45.648962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" event={"ID":"8beb5ce5-17f9-4332-a248-33241e5ab02a","Type":"ContainerStarted","Data":"04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d"}
Jul 09 18:14:45.649373 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:45.649041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" event={"ID":"8beb5ce5-17f9-4332-a248-33241e5ab02a","Type":"ContainerStarted","Data":"ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb"}
Jul 09 18:14:45.649373 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:45.649256    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:45.671912 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:45.671876    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podStartSLOduration=2.861520964 podStartE2EDuration="6.67186189s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.810340926s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:45.670757259 +0000 UTC m=+658.120339704" watchObservedRunningTime="2026-07-09 18:14:45.67186189 +0000 UTC m=+658.121444313"
Jul 09 18:14:46.653853 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:46.653818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:46.655051 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:46.655025    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:14:47.658521 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:47.658483    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:14:48.295575 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:48.295531    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 09 18:14:50.302206 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:50.302163    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:14:50.302684 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:50.302564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:52.663339 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:52.663311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:14:52.663847 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:52.663821    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:14:53.295639 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:53.295598    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 09 18:14:53.295808 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:53.295718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:14:58.294887 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:14:58.294842    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 09 18:15:00.302750 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:00.302703    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:15:00.303168 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:00.302872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:15:00.303168 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:00.302942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:00.303168 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:00.303017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:15:02.664284 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:02.664244    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:15:03.294705 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:03.294654    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 09 18:15:08.295571 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:08.295515    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 09 18:15:09.595745 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.595718    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:15:09.692603 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.692573    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-cabundle-cert\") pod \"37832215-3d18-429f-829d-757ba6479270\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") "
Jul 09 18:15:09.692772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.692620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-kube-rbac-proxy-sar-config\") pod \"37832215-3d18-429f-829d-757ba6479270\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") "
Jul 09 18:15:09.692772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.692660    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpdkk\" (UniqueName: \"kubernetes.io/projected/37832215-3d18-429f-829d-757ba6479270-kube-api-access-jpdkk\") pod \"37832215-3d18-429f-829d-757ba6479270\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") "
Jul 09 18:15:09.692772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.692706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37832215-3d18-429f-829d-757ba6479270-proxy-tls\") pod \"37832215-3d18-429f-829d-757ba6479270\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") "
Jul 09 18:15:09.692772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.692763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37832215-3d18-429f-829d-757ba6479270-kserve-provision-location\") pod \"37832215-3d18-429f-829d-757ba6479270\" (UID: \"37832215-3d18-429f-829d-757ba6479270\") "
Jul 09 18:15:09.693002 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.692964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37832215-3d18-429f-829d-757ba6479270" (UID: "37832215-3d18-429f-829d-757ba6479270"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:09.693150 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.693118    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37832215-3d18-429f-829d-757ba6479270" (UID: "37832215-3d18-429f-829d-757ba6479270"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:09.693150 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.693138    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37832215-3d18-429f-829d-757ba6479270-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37832215-3d18-429f-829d-757ba6479270" (UID: "37832215-3d18-429f-829d-757ba6479270"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:15:09.694840 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.694816    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37832215-3d18-429f-829d-757ba6479270-kube-api-access-jpdkk" (OuterVolumeSpecName: "kube-api-access-jpdkk") pod "37832215-3d18-429f-829d-757ba6479270" (UID: "37832215-3d18-429f-829d-757ba6479270"). InnerVolumeSpecName "kube-api-access-jpdkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:09.694922 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.694839    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37832215-3d18-429f-829d-757ba6479270-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37832215-3d18-429f-829d-757ba6479270" (UID: "37832215-3d18-429f-829d-757ba6479270"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:15:09.750184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.750109    2563 generic.go:356] "Generic (PLEG): container finished" podID="37832215-3d18-429f-829d-757ba6479270" containerID="db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0" exitCode=0
Jul 09 18:15:09.750184 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.750165    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"
Jul 09 18:15:09.750339 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.750194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerDied","Data":"db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0"}
Jul 09 18:15:09.750339 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.750261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv" event={"ID":"37832215-3d18-429f-829d-757ba6479270","Type":"ContainerDied","Data":"24fd6beacc317ac36457eac774f4c3daf209f746bff544e8b788742b4d59a209"}
Jul 09 18:15:09.750339 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.750292    2563 scope.go:117] "RemoveContainer" containerID="db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0"
Jul 09 18:15:09.763716 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.760606    2563 scope.go:117] "RemoveContainer" containerID="33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794"
Jul 09 18:15:09.772987 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.772965    2563 scope.go:117] "RemoveContainer" containerID="9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4"
Jul 09 18:15:09.773457 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.773439    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"]
Jul 09 18:15:09.779419 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.779383    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-c3a93-predictor-5b48894968-62hlv"]
Jul 09 18:15:09.780733 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.780705    2563 scope.go:117] "RemoveContainer" containerID="a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42"
Jul 09 18:15:09.788047 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.788021    2563 scope.go:117] "RemoveContainer" containerID="db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0"
Jul 09 18:15:09.788285 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:15:09.788269    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0\": container with ID starting with db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0 not found: ID does not exist" containerID="db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0"
Jul 09 18:15:09.788341 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.788293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0"} err="failed to get container status \"db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0\": rpc error: code = NotFound desc = could not find container \"db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0\": container with ID starting with db8a05006664e5b97964ab5385340af09b1181cdc0916038964cb5802c7a51d0 not found: ID does not exist"
Jul 09 18:15:09.788341 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.788309    2563 scope.go:117] "RemoveContainer" containerID="33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794"
Jul 09 18:15:09.788587 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:15:09.788572    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794\": container with ID starting with 33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794 not found: ID does not exist" containerID="33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794"
Jul 09 18:15:09.788640 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.788591    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794"} err="failed to get container status \"33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794\": rpc error: code = NotFound desc = could not find container \"33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794\": container with ID starting with 33463562568bb139afbe3373e3e7b269be5aa1cae9a948f73ab5a232d3fd1794 not found: ID does not exist"
Jul 09 18:15:09.788640 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.788604    2563 scope.go:117] "RemoveContainer" containerID="9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4"
Jul 09 18:15:09.788871 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:15:09.788853    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4\": container with ID starting with 9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4 not found: ID does not exist" containerID="9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4"
Jul 09 18:15:09.788932 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.788880    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4"} err="failed to get container status \"9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4\": rpc error: code = NotFound desc = could not find container \"9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4\": container with ID starting with 9f69cca7c8bbc8e7a5e8e20d1bd4fc258d0f1792594be2bbb514b424d0a18da4 not found: ID does not exist"
Jul 09 18:15:09.788932 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.788902    2563 scope.go:117] "RemoveContainer" containerID="a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42"
Jul 09 18:15:09.789129 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:15:09.789114    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42\": container with ID starting with a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42 not found: ID does not exist" containerID="a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42"
Jul 09 18:15:09.789175 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.789133    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42"} err="failed to get container status \"a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42\": rpc error: code = NotFound desc = could not find container \"a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42\": container with ID starting with a7c120474fcf6e2135b730b52605af16174dc713858a1858fc71b7633e5dab42 not found: ID does not exist"
Jul 09 18:15:09.793530 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.793511    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:15:09.793615 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.793535    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37832215-3d18-429f-829d-757ba6479270-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:15:09.793615 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.793550    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdkk\" (UniqueName: \"kubernetes.io/projected/37832215-3d18-429f-829d-757ba6479270-kube-api-access-jpdkk\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:15:09.793615 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.793563    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37832215-3d18-429f-829d-757ba6479270-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:15:09.793615 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.793576    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37832215-3d18-429f-829d-757ba6479270-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:15:09.972394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:09.972364    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37832215-3d18-429f-829d-757ba6479270" path="/var/lib/kubelet/pods/37832215-3d18-429f-829d-757ba6479270/volumes"
Jul 09 18:15:12.663976 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:12.663931    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:15:22.663803 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:22.663760    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:15:32.664623 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:32.664577    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:15:42.664362 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:42.664324    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:15:52.663905 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:15:52.663863    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:16:02.664785 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:02.664745    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:16:29.849809 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.849779    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"]
Jul 09 18:16:29.850335 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.850016    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" containerID="cri-o://ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb" gracePeriod=30
Jul 09 18:16:29.850335 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.850075    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kube-rbac-proxy" containerID="cri-o://04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d" gracePeriod=30
Jul 09 18:16:29.879795 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:16:29.879759    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8beb5ce5_17f9_4332_a248_33241e5ab02a.slice/crio-conmon-04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8beb5ce5_17f9_4332_a248_33241e5ab02a.slice/crio-04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d.scope\": RecentStats: unable to find data in memory cache]"
Jul 09 18:16:29.927721 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.927690    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"]
Jul 09 18:16:29.928017 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928004    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent"
Jul 09 18:16:29.928078 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928018    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent"
Jul 09 18:16:29.928078 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928033    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy"
Jul 09 18:16:29.928078 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928038    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy"
Jul 09 18:16:29.928078 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928058    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="storage-initializer"
Jul 09 18:16:29.928078 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928065    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="storage-initializer"
Jul 09 18:16:29.928078 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928072    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container"
Jul 09 18:16:29.928078 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928078    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container"
Jul 09 18:16:29.928423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928125    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kube-rbac-proxy"
Jul 09 18:16:29.928423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928138    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="agent"
Jul 09 18:16:29.928423 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.928145    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37832215-3d18-429f-829d-757ba6479270" containerName="kserve-container"
Jul 09 18:16:29.931253 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.931227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:29.933981 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.933960    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-a483d-predictor-serving-cert\""
Jul 09 18:16:29.934096 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.934014    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-a483d-kube-rbac-proxy-sar-config\""
Jul 09 18:16:29.943526 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.943501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:29.943649 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.943578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4f62e3-3be5-41db-8260-afccbf844777-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:29.943649 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.943611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4f62e3-3be5-41db-8260-afccbf844777-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:29.943649 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.943634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthvq\" (UniqueName: \"kubernetes.io/projected/cb4f62e3-3be5-41db-8260-afccbf844777-kube-api-access-zthvq\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:29.943800 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.943686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:29.944069 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:29.944052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"]
Jul 09 18:16:30.044581 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.044536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4f62e3-3be5-41db-8260-afccbf844777-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.044755 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.044596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4f62e3-3be5-41db-8260-afccbf844777-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.044755 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.044625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthvq\" (UniqueName: \"kubernetes.io/projected/cb4f62e3-3be5-41db-8260-afccbf844777-kube-api-access-zthvq\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.044755 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.044690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.044755 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.044737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.044995 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.044971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4f62e3-3be5-41db-8260-afccbf844777-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.045616 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.045593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.045723 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.045681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.047338 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.047319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4f62e3-3be5-41db-8260-afccbf844777-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.052575 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.052547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthvq\" (UniqueName: \"kubernetes.io/projected/cb4f62e3-3be5-41db-8260-afccbf844777-kube-api-access-zthvq\") pod \"isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.079586 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.079555    2563 generic.go:356] "Generic (PLEG): container finished" podID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerID="04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d" exitCode=2
Jul 09 18:16:30.079732 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.079673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" event={"ID":"8beb5ce5-17f9-4332-a248-33241e5ab02a","Type":"ContainerDied","Data":"04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d"}
Jul 09 18:16:30.244588 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.244555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:30.404001 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:30.403974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"]
Jul 09 18:16:30.404705 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:16:30.404680    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4f62e3_3be5_41db_8260_afccbf844777.slice/crio-b5fcd93f1982407208e6376769d36bfca1bc5bdd745fa5aa38c1d925d5603d06 WatchSource:0}: Error finding container b5fcd93f1982407208e6376769d36bfca1bc5bdd745fa5aa38c1d925d5603d06: Status 404 returned error can't find the container with id b5fcd93f1982407208e6376769d36bfca1bc5bdd745fa5aa38c1d925d5603d06
Jul 09 18:16:31.086056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:31.086017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" event={"ID":"cb4f62e3-3be5-41db-8260-afccbf844777","Type":"ContainerStarted","Data":"f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d"}
Jul 09 18:16:31.086484 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:31.086100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" event={"ID":"cb4f62e3-3be5-41db-8260-afccbf844777","Type":"ContainerStarted","Data":"b5fcd93f1982407208e6376769d36bfca1bc5bdd745fa5aa38c1d925d5603d06"}
Jul 09 18:16:32.659623 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:32.659577    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 09 18:16:32.663925 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:32.663899    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:16:34.099665 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.099633    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb4f62e3-3be5-41db-8260-afccbf844777" containerID="f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d" exitCode=0
Jul 09 18:16:34.100030 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.099711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" event={"ID":"cb4f62e3-3be5-41db-8260-afccbf844777","Type":"ContainerDied","Data":"f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d"}
Jul 09 18:16:34.596645 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.596619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:16:34.675320 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.675292    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls\") pod \"8beb5ce5-17f9-4332-a248-33241e5ab02a\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") "
Jul 09 18:16:34.675504 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.675353    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beb5ce5-17f9-4332-a248-33241e5ab02a-kserve-provision-location\") pod \"8beb5ce5-17f9-4332-a248-33241e5ab02a\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") "
Jul 09 18:16:34.675504 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.675377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8vbc\" (UniqueName: \"kubernetes.io/projected/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-api-access-r8vbc\") pod \"8beb5ce5-17f9-4332-a248-33241e5ab02a\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") "
Jul 09 18:16:34.675504 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.675416    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-cabundle-cert\") pod \"8beb5ce5-17f9-4332-a248-33241e5ab02a\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") "
Jul 09 18:16:34.675504 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.675439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-rbac-proxy-sar-config\") pod \"8beb5ce5-17f9-4332-a248-33241e5ab02a\" (UID: \"8beb5ce5-17f9-4332-a248-33241e5ab02a\") "
Jul 09 18:16:34.675764 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.675732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beb5ce5-17f9-4332-a248-33241e5ab02a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8beb5ce5-17f9-4332-a248-33241e5ab02a" (UID: "8beb5ce5-17f9-4332-a248-33241e5ab02a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:34.675877 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.675732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8beb5ce5-17f9-4332-a248-33241e5ab02a" (UID: "8beb5ce5-17f9-4332-a248-33241e5ab02a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:34.675877 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.675821    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8beb5ce5-17f9-4332-a248-33241e5ab02a" (UID: "8beb5ce5-17f9-4332-a248-33241e5ab02a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:34.677553 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.677525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-api-access-r8vbc" (OuterVolumeSpecName: "kube-api-access-r8vbc") pod "8beb5ce5-17f9-4332-a248-33241e5ab02a" (UID: "8beb5ce5-17f9-4332-a248-33241e5ab02a"). InnerVolumeSpecName "kube-api-access-r8vbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:34.677634 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.677588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8beb5ce5-17f9-4332-a248-33241e5ab02a" (UID: "8beb5ce5-17f9-4332-a248-33241e5ab02a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:16:34.776538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.776456    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beb5ce5-17f9-4332-a248-33241e5ab02a-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:16:34.776538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.776482    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vbc\" (UniqueName: \"kubernetes.io/projected/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-api-access-r8vbc\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:16:34.776538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.776491    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:16:34.776538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.776501    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8beb5ce5-17f9-4332-a248-33241e5ab02a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:16:34.776538 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:34.776515    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8beb5ce5-17f9-4332-a248-33241e5ab02a-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:16:35.106090 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.106004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" event={"ID":"cb4f62e3-3be5-41db-8260-afccbf844777","Type":"ContainerStarted","Data":"1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416"}
Jul 09 18:16:35.106090 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.106074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" event={"ID":"cb4f62e3-3be5-41db-8260-afccbf844777","Type":"ContainerStarted","Data":"44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3"}
Jul 09 18:16:35.106671 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.106288    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:35.106671 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.106341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:35.107957 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.107934    2563 generic.go:356] "Generic (PLEG): container finished" podID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerID="ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb" exitCode=0
Jul 09 18:16:35.108073 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.107990    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"
Jul 09 18:16:35.108073 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.108011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" event={"ID":"8beb5ce5-17f9-4332-a248-33241e5ab02a","Type":"ContainerDied","Data":"ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb"}
Jul 09 18:16:35.108073 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.108061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj" event={"ID":"8beb5ce5-17f9-4332-a248-33241e5ab02a","Type":"ContainerDied","Data":"14ff6e5a97d948cd2aef199c332872ab9609a71552baa03334a1641b6721c76d"}
Jul 09 18:16:35.108182 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.108087    2563 scope.go:117] "RemoveContainer" containerID="04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d"
Jul 09 18:16:35.108182 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.108093    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:16:35.116963 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.116844    2563 scope.go:117] "RemoveContainer" containerID="ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb"
Jul 09 18:16:35.123908 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.123893    2563 scope.go:117] "RemoveContainer" containerID="0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37"
Jul 09 18:16:35.127662 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.127626    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podStartSLOduration=2.722105365 podStartE2EDuration="6.127612998s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.405507633s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:35.126628697 +0000 UTC m=+767.576211149" watchObservedRunningTime="2026-07-09 18:16:35.127612998 +0000 UTC m=+767.577195422"
Jul 09 18:16:35.131757 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.131739    2563 scope.go:117] "RemoveContainer" containerID="04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d"
Jul 09 18:16:35.132003 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:16:35.131987    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d\": container with ID starting with 04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d not found: ID does not exist" containerID="04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d"
Jul 09 18:16:35.132057 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.132011    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d"} err="failed to get container status \"04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d\": rpc error: code = NotFound desc = could not find container \"04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d\": container with ID starting with 04a0b52007774e262ab22a91b2a6a232ed065990f2247d7671050dda5b68852d not found: ID does not exist"
Jul 09 18:16:35.132057 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.132028    2563 scope.go:117] "RemoveContainer" containerID="ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb"
Jul 09 18:16:35.132276 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:16:35.132255    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb\": container with ID starting with ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb not found: ID does not exist" containerID="ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb"
Jul 09 18:16:35.132359 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.132281    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb"} err="failed to get container status \"ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb\": rpc error: code = NotFound desc = could not find container \"ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb\": container with ID starting with ec055fbbe2c0ccaaf064a9cbe71a3e2114428ac01ea416022fc25d74df7951cb not found: ID does not exist"
Jul 09 18:16:35.132359 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.132296    2563 scope.go:117] "RemoveContainer" containerID="0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37"
Jul 09 18:16:35.132554 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:16:35.132536    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37\": container with ID starting with 0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37 not found: ID does not exist" containerID="0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37"
Jul 09 18:16:35.132594 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.132561    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37"} err="failed to get container status \"0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37\": rpc error: code = NotFound desc = could not find container \"0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37\": container with ID starting with 0890ec352b761b205d44192def97899030091d90ee6cbe7d9775e95e43581b37 not found: ID does not exist"
Jul 09 18:16:35.139860 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.139840    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"]
Jul 09 18:16:35.149177 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.147620    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a2b25-predictor-795794b9cb-2znlj"]
Jul 09 18:16:35.972104 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:35.972065    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" path="/var/lib/kubelet/pods/8beb5ce5-17f9-4332-a248-33241e5ab02a/volumes"
Jul 09 18:16:36.113482 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:36.113445    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:16:41.117688 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:41.117656    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:16:41.118293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:41.118266    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:16:51.118326 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:16:51.118288    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:17:01.118742 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:17:01.118700    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:17:11.118557 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:17:11.118460    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:17:21.119222 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:17:21.119179    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:17:31.118496 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:17:31.118454    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:17:41.118581 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:17:41.118540    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:17:51.119310 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:17:51.119278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:18:20.238834 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.238803    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"]
Jul 09 18:18:20.239419 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.239032    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" containerID="cri-o://44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3" gracePeriod=30
Jul 09 18:18:20.239419 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.239110    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kube-rbac-proxy" containerID="cri-o://1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416" gracePeriod=30
Jul 09 18:18:20.283293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283265    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"]
Jul 09 18:18:20.283718 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283704    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kube-rbac-proxy"
Jul 09 18:18:20.283772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283722    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kube-rbac-proxy"
Jul 09 18:18:20.283772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container"
Jul 09 18:18:20.283772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container"
Jul 09 18:18:20.283772 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283764    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="storage-initializer"
Jul 09 18:18:20.283908 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283773    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="storage-initializer"
Jul 09 18:18:20.283908 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283847    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kube-rbac-proxy"
Jul 09 18:18:20.283908 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.283859    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8beb5ce5-17f9-4332-a248-33241e5ab02a" containerName="kserve-container"
Jul 09 18:18:20.287107 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.287092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.289375 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.289354    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-19737-predictor-serving-cert\""
Jul 09 18:18:20.289690 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.289674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-19737-kube-rbac-proxy-sar-config\""
Jul 09 18:18:20.296676 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.296654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"]
Jul 09 18:18:20.390294 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.390262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxhb\" (UniqueName: \"kubernetes.io/projected/b7957873-fb1c-404c-adf3-b256815b34d1-kube-api-access-wfxhb\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.390447 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.390320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7957873-fb1c-404c-adf3-b256815b34d1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.390447 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.390349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7957873-fb1c-404c-adf3-b256815b34d1-proxy-tls\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.491396 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.491322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxhb\" (UniqueName: \"kubernetes.io/projected/b7957873-fb1c-404c-adf3-b256815b34d1-kube-api-access-wfxhb\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.491396 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.491383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7957873-fb1c-404c-adf3-b256815b34d1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.491586 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.491422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7957873-fb1c-404c-adf3-b256815b34d1-proxy-tls\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.492063 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.492041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7957873-fb1c-404c-adf3-b256815b34d1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.493670 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.493652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7957873-fb1c-404c-adf3-b256815b34d1-proxy-tls\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.499774 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.499754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxhb\" (UniqueName: \"kubernetes.io/projected/b7957873-fb1c-404c-adf3-b256815b34d1-kube-api-access-wfxhb\") pod \"message-dumper-raw-19737-predictor-b86898965-dcf6x\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") " pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.525454 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.525434    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb4f62e3-3be5-41db-8260-afccbf844777" containerID="1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416" exitCode=2
Jul 09 18:18:20.525546 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.525491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" event={"ID":"cb4f62e3-3be5-41db-8260-afccbf844777","Type":"ContainerDied","Data":"1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416"}
Jul 09 18:18:20.596862 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.596836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:20.726500 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.726473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"]
Jul 09 18:18:20.727987 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:18:20.727963    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7957873_fb1c_404c_adf3_b256815b34d1.slice/crio-2496dc17a76cc222dee6908d91c4b4f1d0347a6a6547737f2c62b0deace07b0e WatchSource:0}: Error finding container 2496dc17a76cc222dee6908d91c4b4f1d0347a6a6547737f2c62b0deace07b0e: Status 404 returned error can't find the container with id 2496dc17a76cc222dee6908d91c4b4f1d0347a6a6547737f2c62b0deace07b0e
Jul 09 18:18:20.729824 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:20.729809    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:18:21.114078 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:21.114040    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 09 18:18:21.118616 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:21.118595    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:18:21.531381 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:21.531344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" event={"ID":"b7957873-fb1c-404c-adf3-b256815b34d1","Type":"ContainerStarted","Data":"2496dc17a76cc222dee6908d91c4b4f1d0347a6a6547737f2c62b0deace07b0e"}
Jul 09 18:18:22.537430 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:22.537372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" event={"ID":"b7957873-fb1c-404c-adf3-b256815b34d1","Type":"ContainerStarted","Data":"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d"}
Jul 09 18:18:22.537852 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:22.537464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" event={"ID":"b7957873-fb1c-404c-adf3-b256815b34d1","Type":"ContainerStarted","Data":"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989"}
Jul 09 18:18:22.537852 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:22.537493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:22.537852 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:22.537513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:22.539465 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:22.539444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:22.558296 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:22.558254    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" podStartSLOduration=1.22674078 podStartE2EDuration="2.55823988s" totalImagesPullingTime="1.3314991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:22.556153697 +0000 UTC m=+875.005736139" watchObservedRunningTime="2026-07-09 18:18:22.55823988 +0000 UTC m=+875.007822359"
Jul 09 18:18:24.682417 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.682382    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:18:24.825245 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.825154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4f62e3-3be5-41db-8260-afccbf844777-proxy-tls\") pod \"cb4f62e3-3be5-41db-8260-afccbf844777\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") "
Jul 09 18:18:24.825245 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.825204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-kube-rbac-proxy-sar-config\") pod \"cb4f62e3-3be5-41db-8260-afccbf844777\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") "
Jul 09 18:18:24.825245 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.825243    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zthvq\" (UniqueName: \"kubernetes.io/projected/cb4f62e3-3be5-41db-8260-afccbf844777-kube-api-access-zthvq\") pod \"cb4f62e3-3be5-41db-8260-afccbf844777\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") "
Jul 09 18:18:24.825547 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.825285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4f62e3-3be5-41db-8260-afccbf844777-kserve-provision-location\") pod \"cb4f62e3-3be5-41db-8260-afccbf844777\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") "
Jul 09 18:18:24.825547 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.825343    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-cabundle-cert\") pod \"cb4f62e3-3be5-41db-8260-afccbf844777\" (UID: \"cb4f62e3-3be5-41db-8260-afccbf844777\") "
Jul 09 18:18:24.825679 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.825649    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4f62e3-3be5-41db-8260-afccbf844777-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb4f62e3-3be5-41db-8260-afccbf844777" (UID: "cb4f62e3-3be5-41db-8260-afccbf844777"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:24.825740 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.825707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb4f62e3-3be5-41db-8260-afccbf844777" (UID: "cb4f62e3-3be5-41db-8260-afccbf844777"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:24.825785 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.825745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb4f62e3-3be5-41db-8260-afccbf844777" (UID: "cb4f62e3-3be5-41db-8260-afccbf844777"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:24.827211 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.827185    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb4f62e3-3be5-41db-8260-afccbf844777-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb4f62e3-3be5-41db-8260-afccbf844777" (UID: "cb4f62e3-3be5-41db-8260-afccbf844777"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:24.827299 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.827242    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4f62e3-3be5-41db-8260-afccbf844777-kube-api-access-zthvq" (OuterVolumeSpecName: "kube-api-access-zthvq") pod "cb4f62e3-3be5-41db-8260-afccbf844777" (UID: "cb4f62e3-3be5-41db-8260-afccbf844777"). InnerVolumeSpecName "kube-api-access-zthvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:24.926771 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.926727    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:18:24.926771 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.926768    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4f62e3-3be5-41db-8260-afccbf844777-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:18:24.926771 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.926780    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4f62e3-3be5-41db-8260-afccbf844777-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:18:24.926985 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.926789    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zthvq\" (UniqueName: \"kubernetes.io/projected/cb4f62e3-3be5-41db-8260-afccbf844777-kube-api-access-zthvq\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:18:24.926985 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:24.926798    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4f62e3-3be5-41db-8260-afccbf844777-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:18:25.551858 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.551827    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb4f62e3-3be5-41db-8260-afccbf844777" containerID="44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3" exitCode=0
Jul 09 18:18:25.552101 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.551886    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"
Jul 09 18:18:25.552101 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.551886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" event={"ID":"cb4f62e3-3be5-41db-8260-afccbf844777","Type":"ContainerDied","Data":"44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3"}
Jul 09 18:18:25.552101 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.552013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n" event={"ID":"cb4f62e3-3be5-41db-8260-afccbf844777","Type":"ContainerDied","Data":"b5fcd93f1982407208e6376769d36bfca1bc5bdd745fa5aa38c1d925d5603d06"}
Jul 09 18:18:25.552101 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.552041    2563 scope.go:117] "RemoveContainer" containerID="1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416"
Jul 09 18:18:25.560724 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.560710    2563 scope.go:117] "RemoveContainer" containerID="44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3"
Jul 09 18:18:25.567729 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.567709    2563 scope.go:117] "RemoveContainer" containerID="f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d"
Jul 09 18:18:25.573656 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.573636    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"]
Jul 09 18:18:25.574795 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.574778    2563 scope.go:117] "RemoveContainer" containerID="1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416"
Jul 09 18:18:25.575023 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:18:25.575007    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416\": container with ID starting with 1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416 not found: ID does not exist" containerID="1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416"
Jul 09 18:18:25.575073 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.575032    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416"} err="failed to get container status \"1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416\": rpc error: code = NotFound desc = could not find container \"1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416\": container with ID starting with 1817e65053928b9ef9597614692df9ad6f36962cd689713cf8ad5c17fc372416 not found: ID does not exist"
Jul 09 18:18:25.575073 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.575047    2563 scope.go:117] "RemoveContainer" containerID="44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3"
Jul 09 18:18:25.575255 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:18:25.575237    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3\": container with ID starting with 44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3 not found: ID does not exist" containerID="44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3"
Jul 09 18:18:25.575318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.575266    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3"} err="failed to get container status \"44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3\": rpc error: code = NotFound desc = could not find container \"44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3\": container with ID starting with 44298d0312e52ab4d6529bc248a3ebe6733111824010516489a98d37cd91cdd3 not found: ID does not exist"
Jul 09 18:18:25.575318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.575288    2563 scope.go:117] "RemoveContainer" containerID="f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d"
Jul 09 18:18:25.575560 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:18:25.575544    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d\": container with ID starting with f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d not found: ID does not exist" containerID="f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d"
Jul 09 18:18:25.575607 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.575563    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d"} err="failed to get container status \"f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d\": rpc error: code = NotFound desc = could not find container \"f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d\": container with ID starting with f6a4e0f500bcd900379966202f5f64839217546d12f983e924c458c9b637bc5d not found: ID does not exist"
Jul 09 18:18:25.578161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.578141    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-a483d-predictor-69cd4f69cb-xvj6n"]
Jul 09 18:18:25.971550 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:25.971519    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" path="/var/lib/kubelet/pods/cb4f62e3-3be5-41db-8260-afccbf844777/volumes"
Jul 09 18:18:29.553213 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:29.553177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:18:30.294370 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294337    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"]
Jul 09 18:18:30.294658 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294645    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kube-rbac-proxy"
Jul 09 18:18:30.294658 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294658    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kube-rbac-proxy"
Jul 09 18:18:30.294783 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294681    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container"
Jul 09 18:18:30.294783 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294689    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container"
Jul 09 18:18:30.294783 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294696    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="storage-initializer"
Jul 09 18:18:30.294783 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="storage-initializer"
Jul 09 18:18:30.294783 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294745    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kserve-container"
Jul 09 18:18:30.294783 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.294754    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4f62e3-3be5-41db-8260-afccbf844777" containerName="kube-rbac-proxy"
Jul 09 18:18:30.299475 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.299448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.302631 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.302605    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-19737-predictor-serving-cert\""
Jul 09 18:18:30.302743 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.302611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:18:30.302743 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.302676    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-19737-kube-rbac-proxy-sar-config\""
Jul 09 18:18:30.303137 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.303115    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:18:30.313610 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.313588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"]
Jul 09 18:18:30.370640 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.370616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-cabundle-cert\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.370780 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.370684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jfj\" (UniqueName: \"kubernetes.io/projected/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-api-access-79jfj\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.370780 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.370721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c88d600c-ef3a-4588-9a7a-644536ea00c6-kserve-provision-location\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.370780 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.370747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.370915 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.370823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.471192 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.471158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c88d600c-ef3a-4588-9a7a-644536ea00c6-kserve-provision-location\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.471192 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.471192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.471433 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.471212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.471433 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.471245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-cabundle-cert\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.471433 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.471296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jfj\" (UniqueName: \"kubernetes.io/projected/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-api-access-79jfj\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.471433 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:18:30.471327    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-serving-cert: secret "isvc-logger-raw-19737-predictor-serving-cert" not found
Jul 09 18:18:30.471433 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:18:30.471424    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls podName:c88d600c-ef3a-4588-9a7a-644536ea00c6 nodeName:}" failed. No retries permitted until 2026-07-09 18:18:30.971385293 +0000 UTC m=+883.420967693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls") pod "isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" (UID: "c88d600c-ef3a-4588-9a7a-644536ea00c6") : secret "isvc-logger-raw-19737-predictor-serving-cert" not found
Jul 09 18:18:30.471719 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.471689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c88d600c-ef3a-4588-9a7a-644536ea00c6-kserve-provision-location\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.472056 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.472039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.472118 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.472098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-cabundle-cert\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.479570 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.479549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jfj\" (UniqueName: \"kubernetes.io/projected/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-api-access-79jfj\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.975645 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.975610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:30.977811 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:30.977791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls\") pod \"isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:31.209317 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:31.209277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:31.340327 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:31.340303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"]
Jul 09 18:18:31.342303 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:18:31.342276    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88d600c_ef3a_4588_9a7a_644536ea00c6.slice/crio-8d0c052bfd42b8a46e84104aa55128716aaebd423e2c62c67117b42d55333825 WatchSource:0}: Error finding container 8d0c052bfd42b8a46e84104aa55128716aaebd423e2c62c67117b42d55333825: Status 404 returned error can't find the container with id 8d0c052bfd42b8a46e84104aa55128716aaebd423e2c62c67117b42d55333825
Jul 09 18:18:31.579811 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:31.579725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerStarted","Data":"c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf"}
Jul 09 18:18:31.579811 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:31.579790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerStarted","Data":"8d0c052bfd42b8a46e84104aa55128716aaebd423e2c62c67117b42d55333825"}
Jul 09 18:18:35.597294 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:35.597258    2563 generic.go:356] "Generic (PLEG): container finished" podID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerID="c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf" exitCode=0
Jul 09 18:18:35.597689 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:35.597331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerDied","Data":"c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf"}
Jul 09 18:18:36.603991 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.603957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerStarted","Data":"7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d"}
Jul 09 18:18:36.604397 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.604022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerStarted","Data":"e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428"}
Jul 09 18:18:36.604397 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.604046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerStarted","Data":"f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e"}
Jul 09 18:18:36.604397 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.604336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:36.604584 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.604427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:36.604584 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.604457    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:36.606037 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.606005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:18:36.606723 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.606703    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:36.627374 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:36.627333    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podStartSLOduration=2.959722433 podStartE2EDuration="6.627319733s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.6675973s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:36.625588323 +0000 UTC m=+889.075170765" watchObservedRunningTime="2026-07-09 18:18:36.627319733 +0000 UTC m=+889.076902156"
Jul 09 18:18:37.609057 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:37.609011    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:18:37.609524 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:37.609499    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:42.613181 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:42.613101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:18:42.613775 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:42.613741    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:18:42.613982 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:42.613951    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:47.903510 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:47.903481    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:18:47.905873 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:47.905854    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:18:52.613721 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:52.613681    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:18:52.616074 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:18:52.614131    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:02.613631 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:02.613584    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:19:02.614050 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:02.614024    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:12.613730 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:12.613693    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:19:12.614185 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:12.614162    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:22.613728 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:22.613682    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:19:22.614272 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:22.614161    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:32.614275 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:32.614224    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:19:32.614729 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:32.614689    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:42.614261 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:42.614217    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:19:42.614846 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:42.614814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:52.614906 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:52.614874    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:19:52.615373 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:19:52.615352    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:20:05.339925 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.339895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-19737-predictor-b86898965-dcf6x_b7957873-fb1c-404c-adf3-b256815b34d1/kserve-container/0.log"
Jul 09 18:20:05.600174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.600086    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"]
Jul 09 18:20:05.600451 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.600384    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" containerID="cri-o://f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e" gracePeriod=30
Jul 09 18:20:05.600629 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.600594    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" containerID="cri-o://7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d" gracePeriod=30
Jul 09 18:20:05.600777 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.600461    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy" containerID="cri-o://e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428" gracePeriod=30
Jul 09 18:20:05.681708 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.681668    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"]
Jul 09 18:20:05.685291 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.685265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.687778 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.687755    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-2aece-kube-rbac-proxy-sar-config\""
Jul 09 18:20:05.688018 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.688002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-2aece-predictor-serving-cert\""
Jul 09 18:20:05.699150 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.699124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"]
Jul 09 18:20:05.744868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.744827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.745022 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.744896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f65bcb-94d2-403d-ad53-1758dc432d2c-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.745022 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.744956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59f65bcb-94d2-403d-ad53-1758dc432d2c-proxy-tls\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.745022 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.744995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528lp\" (UniqueName: \"kubernetes.io/projected/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-api-access-528lp\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.745022 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.745017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-cabundle-cert\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.745503 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.745485    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"]
Jul 09 18:20:05.745713 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.745688    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" containerName="kserve-container" containerID="cri-o://988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989" gracePeriod=30
Jul 09 18:20:05.745765 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.745719    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" containerName="kube-rbac-proxy" containerID="cri-o://7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d" gracePeriod=30
Jul 09 18:20:05.845733 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.845691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.845902 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.845759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f65bcb-94d2-403d-ad53-1758dc432d2c-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.845902 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.845794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59f65bcb-94d2-403d-ad53-1758dc432d2c-proxy-tls\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.845902 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.845822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528lp\" (UniqueName: \"kubernetes.io/projected/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-api-access-528lp\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.845902 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.845843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-cabundle-cert\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.846210 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.846178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f65bcb-94d2-403d-ad53-1758dc432d2c-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.846512 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.846489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-cabundle-cert\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.846693 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.846671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.848306 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.848285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59f65bcb-94d2-403d-ad53-1758dc432d2c-proxy-tls\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.855903 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.855839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-528lp\" (UniqueName: \"kubernetes.io/projected/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-api-access-528lp\") pod \"isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:05.982876 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.982841    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:20:05.987438 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.987395    2563 generic.go:356] "Generic (PLEG): container finished" podID="b7957873-fb1c-404c-adf3-b256815b34d1" containerID="7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d" exitCode=2
Jul 09 18:20:05.987438 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.987435    2563 generic.go:356] "Generic (PLEG): container finished" podID="b7957873-fb1c-404c-adf3-b256815b34d1" containerID="988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989" exitCode=2
Jul 09 18:20:05.987438 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.987435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" event={"ID":"b7957873-fb1c-404c-adf3-b256815b34d1","Type":"ContainerDied","Data":"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d"}
Jul 09 18:20:05.987668 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.987475    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"
Jul 09 18:20:05.987668 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.987499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" event={"ID":"b7957873-fb1c-404c-adf3-b256815b34d1","Type":"ContainerDied","Data":"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989"}
Jul 09 18:20:05.987668 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.987531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x" event={"ID":"b7957873-fb1c-404c-adf3-b256815b34d1","Type":"ContainerDied","Data":"2496dc17a76cc222dee6908d91c4b4f1d0347a6a6547737f2c62b0deace07b0e"}
Jul 09 18:20:05.987668 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.987563    2563 scope.go:117] "RemoveContainer" containerID="7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d"
Jul 09 18:20:05.990130 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.990112    2563 generic.go:356] "Generic (PLEG): container finished" podID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerID="e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428" exitCode=2
Jul 09 18:20:05.990240 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.990184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerDied","Data":"e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428"}
Jul 09 18:20:05.995567 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.995547    2563 scope.go:117] "RemoveContainer" containerID="988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989"
Jul 09 18:20:05.995675 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:05.995625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:06.005063 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.005041    2563 scope.go:117] "RemoveContainer" containerID="7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d"
Jul 09 18:20:06.005368 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:20:06.005339    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d\": container with ID starting with 7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d not found: ID does not exist" containerID="7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d"
Jul 09 18:20:06.005471 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.005377    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d"} err="failed to get container status \"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d\": rpc error: code = NotFound desc = could not find container \"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d\": container with ID starting with 7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d not found: ID does not exist"
Jul 09 18:20:06.005471 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.005396    2563 scope.go:117] "RemoveContainer" containerID="988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989"
Jul 09 18:20:06.005686 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:20:06.005664    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989\": container with ID starting with 988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989 not found: ID does not exist" containerID="988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989"
Jul 09 18:20:06.005784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.005690    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989"} err="failed to get container status \"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989\": rpc error: code = NotFound desc = could not find container \"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989\": container with ID starting with 988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989 not found: ID does not exist"
Jul 09 18:20:06.005784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.005710    2563 scope.go:117] "RemoveContainer" containerID="7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d"
Jul 09 18:20:06.006059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.006030    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d"} err="failed to get container status \"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d\": rpc error: code = NotFound desc = could not find container \"7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d\": container with ID starting with 7dc89d67b80889a91f71170917b0dbe17b829cca1ea7cc485fe3e04654ba8b4d not found: ID does not exist"
Jul 09 18:20:06.006059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.006057    2563 scope.go:117] "RemoveContainer" containerID="988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989"
Jul 09 18:20:06.006383 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.006361    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989"} err="failed to get container status \"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989\": rpc error: code = NotFound desc = could not find container \"988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989\": container with ID starting with 988aea8b1a2f0843034bf9f280f5c49962311ef6d144d4f3db4a7cf1598c1989 not found: ID does not exist"
Jul 09 18:20:06.048059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.047725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7957873-fb1c-404c-adf3-b256815b34d1-kube-rbac-proxy-sar-config\") pod \"b7957873-fb1c-404c-adf3-b256815b34d1\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") "
Jul 09 18:20:06.048059 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.047791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7957873-fb1c-404c-adf3-b256815b34d1-proxy-tls\") pod \"b7957873-fb1c-404c-adf3-b256815b34d1\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") "
Jul 09 18:20:06.048823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.048141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfxhb\" (UniqueName: \"kubernetes.io/projected/b7957873-fb1c-404c-adf3-b256815b34d1-kube-api-access-wfxhb\") pod \"b7957873-fb1c-404c-adf3-b256815b34d1\" (UID: \"b7957873-fb1c-404c-adf3-b256815b34d1\") "
Jul 09 18:20:06.048823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.048290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7957873-fb1c-404c-adf3-b256815b34d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b7957873-fb1c-404c-adf3-b256815b34d1" (UID: "b7957873-fb1c-404c-adf3-b256815b34d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:20:06.048823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.048474    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7957873-fb1c-404c-adf3-b256815b34d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:20:06.051088 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.050992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7957873-fb1c-404c-adf3-b256815b34d1-kube-api-access-wfxhb" (OuterVolumeSpecName: "kube-api-access-wfxhb") pod "b7957873-fb1c-404c-adf3-b256815b34d1" (UID: "b7957873-fb1c-404c-adf3-b256815b34d1"). InnerVolumeSpecName "kube-api-access-wfxhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:20:06.051724 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.051700    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7957873-fb1c-404c-adf3-b256815b34d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7957873-fb1c-404c-adf3-b256815b34d1" (UID: "b7957873-fb1c-404c-adf3-b256815b34d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:20:06.133214 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.133190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"]
Jul 09 18:20:06.136228 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:20:06.136197    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f65bcb_94d2_403d_ad53_1758dc432d2c.slice/crio-c14ff8754b8a33acbd8c8286d9324a5ed9fc1f9412988f7e3a62c9e3fd442721 WatchSource:0}: Error finding container c14ff8754b8a33acbd8c8286d9324a5ed9fc1f9412988f7e3a62c9e3fd442721: Status 404 returned error can't find the container with id c14ff8754b8a33acbd8c8286d9324a5ed9fc1f9412988f7e3a62c9e3fd442721
Jul 09 18:20:06.149564 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.149538    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfxhb\" (UniqueName: \"kubernetes.io/projected/b7957873-fb1c-404c-adf3-b256815b34d1-kube-api-access-wfxhb\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:20:06.149652 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.149568    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7957873-fb1c-404c-adf3-b256815b34d1-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:20:06.308704 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.308672    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"]
Jul 09 18:20:06.312313 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.312288    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-19737-predictor-b86898965-dcf6x"]
Jul 09 18:20:06.996680 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.996644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" event={"ID":"59f65bcb-94d2-403d-ad53-1758dc432d2c","Type":"ContainerStarted","Data":"f2de1880cea087203585dd5787eccc7d9d524915a5b8cc7624760c41da66ced6"}
Jul 09 18:20:06.997077 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:06.996702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" event={"ID":"59f65bcb-94d2-403d-ad53-1758dc432d2c","Type":"ContainerStarted","Data":"c14ff8754b8a33acbd8c8286d9324a5ed9fc1f9412988f7e3a62c9e3fd442721"}
Jul 09 18:20:07.610029 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:07.609985    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 09 18:20:07.971949 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:07.971915    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" path="/var/lib/kubelet/pods/b7957873-fb1c-404c-adf3-b256815b34d1/volumes"
Jul 09 18:20:11.014006 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:11.013977    2563 generic.go:356] "Generic (PLEG): container finished" podID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerID="f2de1880cea087203585dd5787eccc7d9d524915a5b8cc7624760c41da66ced6" exitCode=0
Jul 09 18:20:11.014442 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:11.014054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" event={"ID":"59f65bcb-94d2-403d-ad53-1758dc432d2c","Type":"ContainerDied","Data":"f2de1880cea087203585dd5787eccc7d9d524915a5b8cc7624760c41da66ced6"}
Jul 09 18:20:11.016647 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:11.016568    2563 generic.go:356] "Generic (PLEG): container finished" podID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerID="f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e" exitCode=0
Jul 09 18:20:11.016647 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:11.016627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerDied","Data":"f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e"}
Jul 09 18:20:12.022568 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:12.022536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" event={"ID":"59f65bcb-94d2-403d-ad53-1758dc432d2c","Type":"ContainerStarted","Data":"5d10dfe6694622e8634a6b8e4d6b5304440f62bfb9547d33f292a0dc748d8e2a"}
Jul 09 18:20:12.022940 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:12.022596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" event={"ID":"59f65bcb-94d2-403d-ad53-1758dc432d2c","Type":"ContainerStarted","Data":"688a503c99c27f11169d9b2d913f8d141a3b50120f82026a440f223e66cc1b64"}
Jul 09 18:20:12.022940 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:12.022792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:12.042736 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:12.042680    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podStartSLOduration=3.233438604 podStartE2EDuration="7.042663051s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809224447s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:20:12.041977427 +0000 UTC m=+984.491559846" watchObservedRunningTime="2026-07-09 18:20:12.042663051 +0000 UTC m=+984.492245474"
Jul 09 18:20:12.609653 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:12.609615    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 09 18:20:12.614363 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:12.614331    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:12.615496 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:12.615475    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:20:13.030379 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:13.030339    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:13.031558 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:13.031530    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:20:14.034770 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:14.034721    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:20:17.610041 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:17.609998    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 09 18:20:17.610520 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:17.610114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:20:19.042349 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:19.042317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:20:19.043046 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:19.043011    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:20:22.609501 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:22.609461    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 09 18:20:22.614143 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:22.614113    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:22.615850 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:22.615825    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:20:27.609779 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:27.609734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 09 18:20:29.043706 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:29.043665    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:20:32.609970 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:32.609923    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 09 18:20:32.614580 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:32.614550    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:32.614699 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:32.614658    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:20:32.615767 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:32.615745    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:20:32.615877 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:32.615838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:20:35.749418 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.749379    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:20:35.896720 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.896649    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79jfj\" (UniqueName: \"kubernetes.io/projected/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-api-access-79jfj\") pod \"c88d600c-ef3a-4588-9a7a-644536ea00c6\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") "
Jul 09 18:20:35.896720 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.896679    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls\") pod \"c88d600c-ef3a-4588-9a7a-644536ea00c6\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") "
Jul 09 18:20:35.896720 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.896720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c88d600c-ef3a-4588-9a7a-644536ea00c6-kserve-provision-location\") pod \"c88d600c-ef3a-4588-9a7a-644536ea00c6\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") "
Jul 09 18:20:35.896946 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.896738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-rbac-proxy-sar-config\") pod \"c88d600c-ef3a-4588-9a7a-644536ea00c6\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") "
Jul 09 18:20:35.896946 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.896854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-cabundle-cert\") pod \"c88d600c-ef3a-4588-9a7a-644536ea00c6\" (UID: \"c88d600c-ef3a-4588-9a7a-644536ea00c6\") "
Jul 09 18:20:35.897113 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.897082    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88d600c-ef3a-4588-9a7a-644536ea00c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c88d600c-ef3a-4588-9a7a-644536ea00c6" (UID: "c88d600c-ef3a-4588-9a7a-644536ea00c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:35.897236 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.897135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c88d600c-ef3a-4588-9a7a-644536ea00c6" (UID: "c88d600c-ef3a-4588-9a7a-644536ea00c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:20:35.897236 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.897177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c88d600c-ef3a-4588-9a7a-644536ea00c6" (UID: "c88d600c-ef3a-4588-9a7a-644536ea00c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:20:35.898795 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.898777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c88d600c-ef3a-4588-9a7a-644536ea00c6" (UID: "c88d600c-ef3a-4588-9a7a-644536ea00c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:20:35.898878 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.898834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-api-access-79jfj" (OuterVolumeSpecName: "kube-api-access-79jfj") pod "c88d600c-ef3a-4588-9a7a-644536ea00c6" (UID: "c88d600c-ef3a-4588-9a7a-644536ea00c6"). InnerVolumeSpecName "kube-api-access-79jfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:20:35.997653 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.997626    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79jfj\" (UniqueName: \"kubernetes.io/projected/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-api-access-79jfj\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:20:35.997653 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.997648    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c88d600c-ef3a-4588-9a7a-644536ea00c6-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:20:35.997784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.997661    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c88d600c-ef3a-4588-9a7a-644536ea00c6-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:20:35.997784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.997675    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:20:35.997784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:35.997684    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c88d600c-ef3a-4588-9a7a-644536ea00c6-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:20:36.136224 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.136194    2563 generic.go:356] "Generic (PLEG): container finished" podID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerID="7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d" exitCode=0
Jul 09 18:20:36.136389 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.136275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"
Jul 09 18:20:36.136389 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.136274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerDied","Data":"7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d"}
Jul 09 18:20:36.136389 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.136360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4" event={"ID":"c88d600c-ef3a-4588-9a7a-644536ea00c6","Type":"ContainerDied","Data":"8d0c052bfd42b8a46e84104aa55128716aaebd423e2c62c67117b42d55333825"}
Jul 09 18:20:36.136576 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.136424    2563 scope.go:117] "RemoveContainer" containerID="7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d"
Jul 09 18:20:36.145701 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.145669    2563 scope.go:117] "RemoveContainer" containerID="e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428"
Jul 09 18:20:36.153112 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.153015    2563 scope.go:117] "RemoveContainer" containerID="f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e"
Jul 09 18:20:36.155033 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.155014    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"]
Jul 09 18:20:36.160267 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.160250    2563 scope.go:117] "RemoveContainer" containerID="c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf"
Jul 09 18:20:36.161549 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.161530    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-19737-predictor-6d4cd5f99b-gxbf4"]
Jul 09 18:20:36.169660 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.169648    2563 scope.go:117] "RemoveContainer" containerID="7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d"
Jul 09 18:20:36.169895 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:20:36.169875    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d\": container with ID starting with 7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d not found: ID does not exist" containerID="7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d"
Jul 09 18:20:36.169952 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.169908    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d"} err="failed to get container status \"7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d\": rpc error: code = NotFound desc = could not find container \"7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d\": container with ID starting with 7a16a24ac74a1b70c225b0edf2833021ddb4c2893ea3a949180dcc81609a068d not found: ID does not exist"
Jul 09 18:20:36.169952 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.169927    2563 scope.go:117] "RemoveContainer" containerID="e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428"
Jul 09 18:20:36.170159 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:20:36.170143    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428\": container with ID starting with e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428 not found: ID does not exist" containerID="e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428"
Jul 09 18:20:36.170201 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.170164    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428"} err="failed to get container status \"e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428\": rpc error: code = NotFound desc = could not find container \"e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428\": container with ID starting with e9fae654c9a90040fbba637b307d174826a83244a5fdaa98c34649f594550428 not found: ID does not exist"
Jul 09 18:20:36.170201 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.170179    2563 scope.go:117] "RemoveContainer" containerID="f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e"
Jul 09 18:20:36.170392 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:20:36.170375    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e\": container with ID starting with f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e not found: ID does not exist" containerID="f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e"
Jul 09 18:20:36.170478 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.170396    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e"} err="failed to get container status \"f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e\": rpc error: code = NotFound desc = could not find container \"f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e\": container with ID starting with f316bfc1d0f8b0389945b925c0502d8e931c69673511767072018da85033e18e not found: ID does not exist"
Jul 09 18:20:36.170478 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.170431    2563 scope.go:117] "RemoveContainer" containerID="c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf"
Jul 09 18:20:36.170654 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:20:36.170639    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf\": container with ID starting with c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf not found: ID does not exist" containerID="c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf"
Jul 09 18:20:36.170692 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:36.170657    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf"} err="failed to get container status \"c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf\": rpc error: code = NotFound desc = could not find container \"c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf\": container with ID starting with c33813d755af8e88cbbf65cd4cf235b11b3ec13a726516b66d9e60d9f80469cf not found: ID does not exist"
Jul 09 18:20:37.971706 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:37.971673    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" path="/var/lib/kubelet/pods/c88d600c-ef3a-4588-9a7a-644536ea00c6/volumes"
Jul 09 18:20:39.043640 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:39.043593    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:20:49.042974 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:49.042928    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:20:59.043234 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:20:59.043180    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:21:09.043874 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:21:09.043822    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:21:19.043934 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:21:19.043897    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:21:29.043242 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:21:29.043194    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:21:39.043287 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:21:39.043246    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:21:45.968265 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:21:45.968161    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:21:55.968721 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:21:55.968680    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:22:05.968825 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:05.968785    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:22:15.968266 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:15.968223    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:22:25.968741 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:25.968699    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:22:35.968749 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:35.968710    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:22:45.972372 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:45.972343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:22:55.832810 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:55.832778    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"]
Jul 09 18:22:55.833310 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:55.833001    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" containerID="cri-o://688a503c99c27f11169d9b2d913f8d141a3b50120f82026a440f223e66cc1b64" gracePeriod=30
Jul 09 18:22:55.833310 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:55.833067    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kube-rbac-proxy" containerID="cri-o://5d10dfe6694622e8634a6b8e4d6b5304440f62bfb9547d33f292a0dc748d8e2a" gracePeriod=30
Jul 09 18:22:55.968795 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:55.968758    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:22:56.678697 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:56.678664    2563 generic.go:356] "Generic (PLEG): container finished" podID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerID="5d10dfe6694622e8634a6b8e4d6b5304440f62bfb9547d33f292a0dc748d8e2a" exitCode=2
Jul 09 18:22:56.678874 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:56.678725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" event={"ID":"59f65bcb-94d2-403d-ad53-1758dc432d2c","Type":"ContainerDied","Data":"5d10dfe6694622e8634a6b8e4d6b5304440f62bfb9547d33f292a0dc748d8e2a"}
Jul 09 18:22:59.035759 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:22:59.035719    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 09 18:23:01.034074 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034039    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"]
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034330    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="storage-initializer"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034342    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="storage-initializer"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034354    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034359    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034366    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034380    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" containerName="kube-rbac-proxy"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034385    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" containerName="kube-rbac-proxy"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034395    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034416    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034427    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" containerName="kserve-container"
Jul 09 18:23:01.034436 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034436    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" containerName="kserve-container"
Jul 09 18:23:01.034823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034492    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" containerName="kube-rbac-proxy"
Jul 09 18:23:01.034823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034502    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kube-rbac-proxy"
Jul 09 18:23:01.034823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034508    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="agent"
Jul 09 18:23:01.034823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034514    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7957873-fb1c-404c-adf3-b256815b34d1" containerName="kserve-container"
Jul 09 18:23:01.034823 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.034520    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88d600c-ef3a-4588-9a7a-644536ea00c6" containerName="kserve-container"
Jul 09 18:23:01.037605 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.037585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.040277 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.040247    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f52a32-kube-rbac-proxy-sar-config\""
Jul 09 18:23:01.040277 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.040246    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f52a32-predictor-serving-cert\""
Jul 09 18:23:01.048784 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.048763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"]
Jul 09 18:23:01.141182 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.141157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d379b307-9d56-4287-b10f-f52336003b8d-proxy-tls\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.141306 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.141203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-cabundle-cert\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.141306 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.141229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.141382 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.141297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl445\" (UniqueName: \"kubernetes.io/projected/d379b307-9d56-4287-b10f-f52336003b8d-kube-api-access-gl445\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.141382 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.141330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d379b307-9d56-4287-b10f-f52336003b8d-kserve-provision-location\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.242145 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.242111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-cabundle-cert\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.242270 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.242159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.242270 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.242194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl445\" (UniqueName: \"kubernetes.io/projected/d379b307-9d56-4287-b10f-f52336003b8d-kube-api-access-gl445\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.242270 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.242219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d379b307-9d56-4287-b10f-f52336003b8d-kserve-provision-location\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.242270 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.242246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d379b307-9d56-4287-b10f-f52336003b8d-proxy-tls\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.242647 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.242623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d379b307-9d56-4287-b10f-f52336003b8d-kserve-provision-location\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.242930 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.242899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-cabundle-cert\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.242991 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.242899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.244658 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.244637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d379b307-9d56-4287-b10f-f52336003b8d-proxy-tls\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.250606 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.250582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl445\" (UniqueName: \"kubernetes.io/projected/d379b307-9d56-4287-b10f-f52336003b8d-kube-api-access-gl445\") pod \"isvc-primary-f52a32-predictor-5b679f957-d5sdz\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") " pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.348038 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.347963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:01.477732 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.477703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"]
Jul 09 18:23:01.480105 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:23:01.480071    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd379b307_9d56_4287_b10f_f52336003b8d.slice/crio-6e48c4b376316de12e4cb18ec527d5d8932ca62938656c3f259687a508fa91f1 WatchSource:0}: Error finding container 6e48c4b376316de12e4cb18ec527d5d8932ca62938656c3f259687a508fa91f1: Status 404 returned error can't find the container with id 6e48c4b376316de12e4cb18ec527d5d8932ca62938656c3f259687a508fa91f1
Jul 09 18:23:01.698845 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.698814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" event={"ID":"d379b307-9d56-4287-b10f-f52336003b8d","Type":"ContainerStarted","Data":"fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b"}
Jul 09 18:23:01.699009 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:01.698872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" event={"ID":"d379b307-9d56-4287-b10f-f52336003b8d","Type":"ContainerStarted","Data":"6e48c4b376316de12e4cb18ec527d5d8932ca62938656c3f259687a508fa91f1"}
Jul 09 18:23:04.035112 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:04.035065    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 09 18:23:05.719817 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.719673    2563 generic.go:356] "Generic (PLEG): container finished" podID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerID="688a503c99c27f11169d9b2d913f8d141a3b50120f82026a440f223e66cc1b64" exitCode=0
Jul 09 18:23:05.720279 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.719997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" event={"ID":"59f65bcb-94d2-403d-ad53-1758dc432d2c","Type":"ContainerDied","Data":"688a503c99c27f11169d9b2d913f8d141a3b50120f82026a440f223e66cc1b64"}
Jul 09 18:23:05.721875 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.721851    2563 generic.go:356] "Generic (PLEG): container finished" podID="d379b307-9d56-4287-b10f-f52336003b8d" containerID="fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b" exitCode=0
Jul 09 18:23:05.722016 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.721885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" event={"ID":"d379b307-9d56-4287-b10f-f52336003b8d","Type":"ContainerDied","Data":"fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b"}
Jul 09 18:23:05.809471 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.809445    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:23:05.980161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.980125    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-528lp\" (UniqueName: \"kubernetes.io/projected/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-api-access-528lp\") pod \"59f65bcb-94d2-403d-ad53-1758dc432d2c\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") "
Jul 09 18:23:05.980358 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.980189    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59f65bcb-94d2-403d-ad53-1758dc432d2c-proxy-tls\") pod \"59f65bcb-94d2-403d-ad53-1758dc432d2c\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") "
Jul 09 18:23:05.980358 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.980223    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-rbac-proxy-sar-config\") pod \"59f65bcb-94d2-403d-ad53-1758dc432d2c\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") "
Jul 09 18:23:05.980358 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.980342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f65bcb-94d2-403d-ad53-1758dc432d2c-kserve-provision-location\") pod \"59f65bcb-94d2-403d-ad53-1758dc432d2c\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") "
Jul 09 18:23:05.980565 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.980386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-cabundle-cert\") pod \"59f65bcb-94d2-403d-ad53-1758dc432d2c\" (UID: \"59f65bcb-94d2-403d-ad53-1758dc432d2c\") "
Jul 09 18:23:05.980687 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.980662    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59f65bcb-94d2-403d-ad53-1758dc432d2c" (UID: "59f65bcb-94d2-403d-ad53-1758dc432d2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:23:05.980750 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.980689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f65bcb-94d2-403d-ad53-1758dc432d2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59f65bcb-94d2-403d-ad53-1758dc432d2c" (UID: "59f65bcb-94d2-403d-ad53-1758dc432d2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:23:05.980837 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.980813    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59f65bcb-94d2-403d-ad53-1758dc432d2c" (UID: "59f65bcb-94d2-403d-ad53-1758dc432d2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:23:05.982374 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.982350    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-api-access-528lp" (OuterVolumeSpecName: "kube-api-access-528lp") pod "59f65bcb-94d2-403d-ad53-1758dc432d2c" (UID: "59f65bcb-94d2-403d-ad53-1758dc432d2c"). InnerVolumeSpecName "kube-api-access-528lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:23:05.982486 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:05.982437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f65bcb-94d2-403d-ad53-1758dc432d2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59f65bcb-94d2-403d-ad53-1758dc432d2c" (UID: "59f65bcb-94d2-403d-ad53-1758dc432d2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:23:06.081894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.081804    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-528lp\" (UniqueName: \"kubernetes.io/projected/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-api-access-528lp\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:23:06.081894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.081838    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59f65bcb-94d2-403d-ad53-1758dc432d2c-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:23:06.081894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.081851    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:23:06.081894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.081859    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f65bcb-94d2-403d-ad53-1758dc432d2c-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:23:06.081894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.081868    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f65bcb-94d2-403d-ad53-1758dc432d2c-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:23:06.727851 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.727806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" event={"ID":"d379b307-9d56-4287-b10f-f52336003b8d","Type":"ContainerStarted","Data":"4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31"}
Jul 09 18:23:06.728293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.727874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" event={"ID":"d379b307-9d56-4287-b10f-f52336003b8d","Type":"ContainerStarted","Data":"bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b"}
Jul 09 18:23:06.728293 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.728163    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:06.728432 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.728311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:06.729794 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.729768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7" event={"ID":"59f65bcb-94d2-403d-ad53-1758dc432d2c","Type":"ContainerDied","Data":"c14ff8754b8a33acbd8c8286d9324a5ed9fc1f9412988f7e3a62c9e3fd442721"}
Jul 09 18:23:06.729794 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.729790    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"
Jul 09 18:23:06.729929 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.729830    2563 scope.go:117] "RemoveContainer" containerID="5d10dfe6694622e8634a6b8e4d6b5304440f62bfb9547d33f292a0dc748d8e2a"
Jul 09 18:23:06.730060 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.730022    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:23:06.739009 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.738991    2563 scope.go:117] "RemoveContainer" containerID="688a503c99c27f11169d9b2d913f8d141a3b50120f82026a440f223e66cc1b64"
Jul 09 18:23:06.749983 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.749687    2563 scope.go:117] "RemoveContainer" containerID="f2de1880cea087203585dd5787eccc7d9d524915a5b8cc7624760c41da66ced6"
Jul 09 18:23:06.750529 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.750482    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podStartSLOduration=2.389737321 podStartE2EDuration="5.750464325s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.360727004s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:23:06.750146598 +0000 UTC m=+1159.199729023" watchObservedRunningTime="2026-07-09 18:23:06.750464325 +0000 UTC m=+1159.200046748"
Jul 09 18:23:06.764308 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.764282    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"]
Jul 09 18:23:06.770474 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:06.770443    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-2aece-predictor-f85bb6978-qphc7"]
Jul 09 18:23:07.735577 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:07.735543    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:23:07.971925 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:07.971890    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" path="/var/lib/kubelet/pods/59f65bcb-94d2-403d-ad53-1758dc432d2c/volumes"
Jul 09 18:23:12.740083 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:12.740001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:23:12.740744 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:12.740713    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:23:22.740699 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:22.740656    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:23:32.740920 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:32.740876    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:23:42.740897 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:42.740854    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:23:47.928090 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:47.928062    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:23:47.929907 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:47.929885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:23:52.740857 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:23:52.740816    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:24:02.740730 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:02.740688    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:24:12.740648 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:12.740610    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:24:17.969762 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:17.969720    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:24:27.972801 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:27.972774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:24:31.228484 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228450    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"]
Jul 09 18:24:31.228868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228760    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container"
Jul 09 18:24:31.228868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228771    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container"
Jul 09 18:24:31.228868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228787    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="storage-initializer"
Jul 09 18:24:31.228868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228792    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="storage-initializer"
Jul 09 18:24:31.228868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228804    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kube-rbac-proxy"
Jul 09 18:24:31.228868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228810    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kube-rbac-proxy"
Jul 09 18:24:31.228868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228857    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kube-rbac-proxy"
Jul 09 18:24:31.228868 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.228869    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f65bcb-94d2-403d-ad53-1758dc432d2c" containerName="kserve-container"
Jul 09 18:24:31.231993 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.231975    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.234618 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.234586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f52a32-predictor-serving-cert\""
Jul 09 18:24:31.234888 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.234864    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-f52a32\""
Jul 09 18:24:31.234976 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.234865    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-f52a32-dockercfg-krx6v\""
Jul 09 18:24:31.234976 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.234872    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f52a32-kube-rbac-proxy-sar-config\""
Jul 09 18:24:31.243932 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.243905    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"]
Jul 09 18:24:31.258107 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.258072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.258284 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.258139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5125bac6-7872-456e-b941-ce518821bdf5-kserve-provision-location\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.258284 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.258168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-cabundle-cert\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.258284 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.258191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zhc\" (UniqueName: \"kubernetes.io/projected/5125bac6-7872-456e-b941-ce518821bdf5-kube-api-access-p5zhc\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.258284 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.258250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.358662 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.358626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5125bac6-7872-456e-b941-ce518821bdf5-kserve-provision-location\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.358875 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.358669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-cabundle-cert\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.358875 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.358709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zhc\" (UniqueName: \"kubernetes.io/projected/5125bac6-7872-456e-b941-ce518821bdf5-kube-api-access-p5zhc\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.358875 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.358742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.359037 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.358879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.359037 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:31.359015    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-serving-cert: secret "isvc-secondary-f52a32-predictor-serving-cert" not found
Jul 09 18:24:31.359138 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:31.359089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls podName:5125bac6-7872-456e-b941-ce518821bdf5 nodeName:}" failed. No retries permitted until 2026-07-09 18:24:31.859068507 +0000 UTC m=+1244.308650923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls") pod "isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5" (UID: "5125bac6-7872-456e-b941-ce518821bdf5") : secret "isvc-secondary-f52a32-predictor-serving-cert" not found
Jul 09 18:24:31.359201 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.359132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5125bac6-7872-456e-b941-ce518821bdf5-kserve-provision-location\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.359486 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.359466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.359554 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.359480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-cabundle-cert\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.367466 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.367433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zhc\" (UniqueName: \"kubernetes.io/projected/5125bac6-7872-456e-b941-ce518821bdf5-kube-api-access-p5zhc\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.862261 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.862225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:31.864656 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:31.864626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls\") pod \"isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") " pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:32.143659 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:32.143551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:32.278083 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:32.278056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"]
Jul 09 18:24:32.280396 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:24:32.280371    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5125bac6_7872_456e_b941_ce518821bdf5.slice/crio-294fd86510eb2874b0d644b0b5fec286c61678ff59596ee97da7ec346db6abce WatchSource:0}: Error finding container 294fd86510eb2874b0d644b0b5fec286c61678ff59596ee97da7ec346db6abce: Status 404 returned error can't find the container with id 294fd86510eb2874b0d644b0b5fec286c61678ff59596ee97da7ec346db6abce
Jul 09 18:24:33.083716 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:33.083681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5" event={"ID":"5125bac6-7872-456e-b941-ce518821bdf5","Type":"ContainerStarted","Data":"ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7"}
Jul 09 18:24:33.083903 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:33.083741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5" event={"ID":"5125bac6-7872-456e-b941-ce518821bdf5","Type":"ContainerStarted","Data":"294fd86510eb2874b0d644b0b5fec286c61678ff59596ee97da7ec346db6abce"}
Jul 09 18:24:38.106696 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:38.106622    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_5125bac6-7872-456e-b941-ce518821bdf5/storage-initializer/0.log"
Jul 09 18:24:38.106696 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:38.106658    2563 generic.go:356] "Generic (PLEG): container finished" podID="5125bac6-7872-456e-b941-ce518821bdf5" containerID="ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7" exitCode=1
Jul 09 18:24:38.106696 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:38.106695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5" event={"ID":"5125bac6-7872-456e-b941-ce518821bdf5","Type":"ContainerDied","Data":"ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7"}
Jul 09 18:24:39.112614 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:39.112587    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_5125bac6-7872-456e-b941-ce518821bdf5/storage-initializer/0.log"
Jul 09 18:24:39.112973 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:39.112663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5" event={"ID":"5125bac6-7872-456e-b941-ce518821bdf5","Type":"ContainerStarted","Data":"cd15fcdda3e6b67cdb81ea49c4a2afefe2e58722df1a3d545195891a1cd38638"}
Jul 09 18:24:44.134687 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:44.134656    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_5125bac6-7872-456e-b941-ce518821bdf5/storage-initializer/1.log"
Jul 09 18:24:44.135247 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:44.135095    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_5125bac6-7872-456e-b941-ce518821bdf5/storage-initializer/0.log"
Jul 09 18:24:44.135247 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:44.135138    2563 generic.go:356] "Generic (PLEG): container finished" podID="5125bac6-7872-456e-b941-ce518821bdf5" containerID="cd15fcdda3e6b67cdb81ea49c4a2afefe2e58722df1a3d545195891a1cd38638" exitCode=1
Jul 09 18:24:44.135247 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:44.135213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5" event={"ID":"5125bac6-7872-456e-b941-ce518821bdf5","Type":"ContainerDied","Data":"cd15fcdda3e6b67cdb81ea49c4a2afefe2e58722df1a3d545195891a1cd38638"}
Jul 09 18:24:44.135375 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:44.135286    2563 scope.go:117] "RemoveContainer" containerID="ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7"
Jul 09 18:24:44.135636 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:44.135620    2563 scope.go:117] "RemoveContainer" containerID="ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7"
Jul 09 18:24:44.146332 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:44.146312    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_kserve-ci-e2e-test(5125bac6-7872-456e-b941-ce518821bdf5)\"" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5" podUID="5125bac6-7872-456e-b941-ce518821bdf5"
Jul 09 18:24:44.146562 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:44.146534    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_kserve-ci-e2e-test_5125bac6-7872-456e-b941-ce518821bdf5_0 in pod sandbox 294fd86510eb2874b0d644b0b5fec286c61678ff59596ee97da7ec346db6abce from index: no such id: 'ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7'" containerID="ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7"
Jul 09 18:24:44.146633 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:44.146572    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_kserve-ci-e2e-test_5125bac6-7872-456e-b941-ce518821bdf5_0 in pod sandbox 294fd86510eb2874b0d644b0b5fec286c61678ff59596ee97da7ec346db6abce from index: no such id: 'ff0b5b14a6e7236c3a3fef71298b3daadce0ab550c62348a0aa5d2fef2d58ea7'"
Jul 09 18:24:45.141189 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:45.141160    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_5125bac6-7872-456e-b941-ce518821bdf5/storage-initializer/1.log"
Jul 09 18:24:49.283588 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.283557    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"]
Jul 09 18:24:49.335563 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.335531    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"]
Jul 09 18:24:49.335804 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.335774    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container" containerID="cri-o://bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b" gracePeriod=30
Jul 09 18:24:49.335870 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.335800    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kube-rbac-proxy" containerID="cri-o://4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31" gracePeriod=30
Jul 09 18:24:49.455315 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.455291    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_5125bac6-7872-456e-b941-ce518821bdf5/storage-initializer/1.log"
Jul 09 18:24:49.457479 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.457463    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:49.501894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.501857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-kube-rbac-proxy-sar-config\") pod \"5125bac6-7872-456e-b941-ce518821bdf5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") "
Jul 09 18:24:49.501894 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.501900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5zhc\" (UniqueName: \"kubernetes.io/projected/5125bac6-7872-456e-b941-ce518821bdf5-kube-api-access-p5zhc\") pod \"5125bac6-7872-456e-b941-ce518821bdf5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") "
Jul 09 18:24:49.502129 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.501939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-cabundle-cert\") pod \"5125bac6-7872-456e-b941-ce518821bdf5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") "
Jul 09 18:24:49.502129 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.501971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls\") pod \"5125bac6-7872-456e-b941-ce518821bdf5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") "
Jul 09 18:24:49.502129 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.502013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5125bac6-7872-456e-b941-ce518821bdf5-kserve-provision-location\") pod \"5125bac6-7872-456e-b941-ce518821bdf5\" (UID: \"5125bac6-7872-456e-b941-ce518821bdf5\") "
Jul 09 18:24:49.502335 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.502296    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5125bac6-7872-456e-b941-ce518821bdf5" (UID: "5125bac6-7872-456e-b941-ce518821bdf5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:49.502470 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.502302    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5125bac6-7872-456e-b941-ce518821bdf5" (UID: "5125bac6-7872-456e-b941-ce518821bdf5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:49.502470 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.502349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5125bac6-7872-456e-b941-ce518821bdf5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5125bac6-7872-456e-b941-ce518821bdf5" (UID: "5125bac6-7872-456e-b941-ce518821bdf5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:49.504045 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.504019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5125bac6-7872-456e-b941-ce518821bdf5-kube-api-access-p5zhc" (OuterVolumeSpecName: "kube-api-access-p5zhc") pod "5125bac6-7872-456e-b941-ce518821bdf5" (UID: "5125bac6-7872-456e-b941-ce518821bdf5"). InnerVolumeSpecName "kube-api-access-p5zhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:49.504115 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.504066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5125bac6-7872-456e-b941-ce518821bdf5" (UID: "5125bac6-7872-456e-b941-ce518821bdf5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:49.603445 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.603330    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5125bac6-7872-456e-b941-ce518821bdf5-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:49.603445 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.603363    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:49.603445 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.603372    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5zhc\" (UniqueName: \"kubernetes.io/projected/5125bac6-7872-456e-b941-ce518821bdf5-kube-api-access-p5zhc\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:49.603445 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.603383    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5125bac6-7872-456e-b941-ce518821bdf5-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:49.603445 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:49.603397    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5125bac6-7872-456e-b941-ce518821bdf5-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:50.164300 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:50.164271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5_5125bac6-7872-456e-b941-ce518821bdf5/storage-initializer/1.log"
Jul 09 18:24:50.164509 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:50.164386    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"
Jul 09 18:24:50.164509 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:50.164434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5" event={"ID":"5125bac6-7872-456e-b941-ce518821bdf5","Type":"ContainerDied","Data":"294fd86510eb2874b0d644b0b5fec286c61678ff59596ee97da7ec346db6abce"}
Jul 09 18:24:50.164639 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:50.164527    2563 scope.go:117] "RemoveContainer" containerID="cd15fcdda3e6b67cdb81ea49c4a2afefe2e58722df1a3d545195891a1cd38638"
Jul 09 18:24:50.166998 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:50.166936    2563 generic.go:356] "Generic (PLEG): container finished" podID="d379b307-9d56-4287-b10f-f52336003b8d" containerID="4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31" exitCode=2
Jul 09 18:24:50.167108 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:50.167006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" event={"ID":"d379b307-9d56-4287-b10f-f52336003b8d","Type":"ContainerDied","Data":"4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31"}
Jul 09 18:24:50.202645 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:50.202611    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"]
Jul 09 18:24:50.206559 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:50.206531    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f52a32-predictor-6f99cfcc78-cvsl5"]
Jul 09 18:24:51.971871 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:51.971837    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5125bac6-7872-456e-b941-ce518821bdf5" path="/var/lib/kubelet/pods/5125bac6-7872-456e-b941-ce518821bdf5/volumes"
Jul 09 18:24:52.736730 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:52.736690    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 09 18:24:53.985432 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:53.985391    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:24:54.041860 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.041791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl445\" (UniqueName: \"kubernetes.io/projected/d379b307-9d56-4287-b10f-f52336003b8d-kube-api-access-gl445\") pod \"d379b307-9d56-4287-b10f-f52336003b8d\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") "
Jul 09 18:24:54.041860 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.041837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-kube-rbac-proxy-sar-config\") pod \"d379b307-9d56-4287-b10f-f52336003b8d\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") "
Jul 09 18:24:54.041860 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.041858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-cabundle-cert\") pod \"d379b307-9d56-4287-b10f-f52336003b8d\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") "
Jul 09 18:24:54.042112 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.041890    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d379b307-9d56-4287-b10f-f52336003b8d-proxy-tls\") pod \"d379b307-9d56-4287-b10f-f52336003b8d\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") "
Jul 09 18:24:54.042112 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.041914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d379b307-9d56-4287-b10f-f52336003b8d-kserve-provision-location\") pod \"d379b307-9d56-4287-b10f-f52336003b8d\" (UID: \"d379b307-9d56-4287-b10f-f52336003b8d\") "
Jul 09 18:24:54.042302 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.042272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379b307-9d56-4287-b10f-f52336003b8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d379b307-9d56-4287-b10f-f52336003b8d" (UID: "d379b307-9d56-4287-b10f-f52336003b8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:54.042363 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.042272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d379b307-9d56-4287-b10f-f52336003b8d" (UID: "d379b307-9d56-4287-b10f-f52336003b8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:54.042363 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.042280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d379b307-9d56-4287-b10f-f52336003b8d" (UID: "d379b307-9d56-4287-b10f-f52336003b8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:54.043861 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.043841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d379b307-9d56-4287-b10f-f52336003b8d-kube-api-access-gl445" (OuterVolumeSpecName: "kube-api-access-gl445") pod "d379b307-9d56-4287-b10f-f52336003b8d" (UID: "d379b307-9d56-4287-b10f-f52336003b8d"). InnerVolumeSpecName "kube-api-access-gl445". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:54.044046 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.044026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d379b307-9d56-4287-b10f-f52336003b8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d379b307-9d56-4287-b10f-f52336003b8d" (UID: "d379b307-9d56-4287-b10f-f52336003b8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:54.142601 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.142553    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl445\" (UniqueName: \"kubernetes.io/projected/d379b307-9d56-4287-b10f-f52336003b8d-kube-api-access-gl445\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:54.142601 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.142584    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:54.142601 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.142594    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d379b307-9d56-4287-b10f-f52336003b8d-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:54.142601 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.142605    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d379b307-9d56-4287-b10f-f52336003b8d-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:54.142865 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.142613    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d379b307-9d56-4287-b10f-f52336003b8d-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:24:54.186363 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.186330    2563 generic.go:356] "Generic (PLEG): container finished" podID="d379b307-9d56-4287-b10f-f52336003b8d" containerID="bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b" exitCode=0
Jul 09 18:24:54.186499 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.186377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" event={"ID":"d379b307-9d56-4287-b10f-f52336003b8d","Type":"ContainerDied","Data":"bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b"}
Jul 09 18:24:54.186499 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.186458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz" event={"ID":"d379b307-9d56-4287-b10f-f52336003b8d","Type":"ContainerDied","Data":"6e48c4b376316de12e4cb18ec527d5d8932ca62938656c3f259687a508fa91f1"}
Jul 09 18:24:54.186499 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.186387    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"
Jul 09 18:24:54.186499 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.186490    2563 scope.go:117] "RemoveContainer" containerID="4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31"
Jul 09 18:24:54.195449 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.195427    2563 scope.go:117] "RemoveContainer" containerID="bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b"
Jul 09 18:24:54.202600 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.202580    2563 scope.go:117] "RemoveContainer" containerID="fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b"
Jul 09 18:24:54.209938 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.209915    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"]
Jul 09 18:24:54.210203 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.210185    2563 scope.go:117] "RemoveContainer" containerID="4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31"
Jul 09 18:24:54.210487 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:54.210471    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31\": container with ID starting with 4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31 not found: ID does not exist" containerID="4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31"
Jul 09 18:24:54.210541 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.210495    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31"} err="failed to get container status \"4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31\": rpc error: code = NotFound desc = could not find container \"4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31\": container with ID starting with 4f0bf4c4d0132af7815f4f9cf5dbb69a7fb7c44c82976fa93939d692e6069f31 not found: ID does not exist"
Jul 09 18:24:54.210541 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.210511    2563 scope.go:117] "RemoveContainer" containerID="bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b"
Jul 09 18:24:54.210724 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:54.210702    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b\": container with ID starting with bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b not found: ID does not exist" containerID="bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b"
Jul 09 18:24:54.210824 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.210729    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b"} err="failed to get container status \"bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b\": rpc error: code = NotFound desc = could not find container \"bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b\": container with ID starting with bb2832b65fb2182ed816c01e9e0a128676739331bccab31f6f2022766c15329b not found: ID does not exist"
Jul 09 18:24:54.210824 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.210746    2563 scope.go:117] "RemoveContainer" containerID="fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b"
Jul 09 18:24:54.211011 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:54.210994    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b\": container with ID starting with fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b not found: ID does not exist" containerID="fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b"
Jul 09 18:24:54.211051 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.211017    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b"} err="failed to get container status \"fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b\": rpc error: code = NotFound desc = could not find container \"fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b\": container with ID starting with fc062f416af7491d254d5d9ddbf34057f1f60b9179def388021f500c4c4d961b not found: ID does not exist"
Jul 09 18:24:54.215032 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.215011    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f52a32-predictor-5b679f957-d5sdz"]
Jul 09 18:24:54.508899 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.508856    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"]
Jul 09 18:24:54.509229 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509211    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kube-rbac-proxy"
Jul 09 18:24:54.509318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509231    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kube-rbac-proxy"
Jul 09 18:24:54.509318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509244    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5125bac6-7872-456e-b941-ce518821bdf5" containerName="storage-initializer"
Jul 09 18:24:54.509318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509252    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5125bac6-7872-456e-b941-ce518821bdf5" containerName="storage-initializer"
Jul 09 18:24:54.509318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509278    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container"
Jul 09 18:24:54.509318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509287    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container"
Jul 09 18:24:54.509318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="storage-initializer"
Jul 09 18:24:54.509318 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509308    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="storage-initializer"
Jul 09 18:24:54.509681 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509386    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kserve-container"
Jul 09 18:24:54.509681 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509397    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d379b307-9d56-4287-b10f-f52336003b8d" containerName="kube-rbac-proxy"
Jul 09 18:24:54.509681 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509426    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5125bac6-7872-456e-b941-ce518821bdf5" containerName="storage-initializer"
Jul 09 18:24:54.509681 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509438    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5125bac6-7872-456e-b941-ce518821bdf5" containerName="storage-initializer"
Jul 09 18:24:54.509681 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5125bac6-7872-456e-b941-ce518821bdf5" containerName="storage-initializer"
Jul 09 18:24:54.509681 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.509537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5125bac6-7872-456e-b941-ce518821bdf5" containerName="storage-initializer"
Jul 09 18:24:54.515232 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.515210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.517617 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.517589    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4d7d87-kube-rbac-proxy-sar-config\""
Jul 09 18:24:54.517787 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.517772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:24:54.517865 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.517815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-4d7d87\""
Jul 09 18:24:54.518036 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.518022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4d7d87-predictor-serving-cert\""
Jul 09 18:24:54.518558 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.518531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-4d7d87-dockercfg-qfdfw\""
Jul 09 18:24:54.518558 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.518543    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:24:54.518558 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.518532    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:24:54.523734 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.523715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"]
Jul 09 18:24:54.646299 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.646267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqm6\" (UniqueName: \"kubernetes.io/projected/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-api-access-gqqm6\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.646497 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.646313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.646497 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.646345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-cabundle-cert\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.646497 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.646436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.646497 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.646464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d7f07-7394-4365-80d0-3d894ce53d19-kserve-provision-location\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.746852 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.746814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-cabundle-cert\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.747021 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.746885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.747021 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.746934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d7f07-7394-4365-80d0-3d894ce53d19-kserve-provision-location\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.747021 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.746972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqm6\" (UniqueName: \"kubernetes.io/projected/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-api-access-gqqm6\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.747149 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.747019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.747149 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:54.747028    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-serving-cert: secret "isvc-init-fail-4d7d87-predictor-serving-cert" not found
Jul 09 18:24:54.747149 ip-10-0-133-76 kubenswrapper[2563]: E0709 18:24:54.747099    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls podName:d49d7f07-7394-4365-80d0-3d894ce53d19 nodeName:}" failed. No retries permitted until 2026-07-09 18:24:55.247077072 +0000 UTC m=+1267.696659487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls") pod "isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd" (UID: "d49d7f07-7394-4365-80d0-3d894ce53d19") : secret "isvc-init-fail-4d7d87-predictor-serving-cert" not found
Jul 09 18:24:54.747421 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.747382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d7f07-7394-4365-80d0-3d894ce53d19-kserve-provision-location\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.747662 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.747642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-cabundle-cert\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.747709 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.747642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:54.757057 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:54.757036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqm6\" (UniqueName: \"kubernetes.io/projected/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-api-access-gqqm6\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:55.249982 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:55.249950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:55.252319 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:55.252300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls\") pod \"isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") " pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:55.426290 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:55.426256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:24:55.555806 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:55.555783    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"]
Jul 09 18:24:55.557618 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:24:55.557592    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49d7f07_7394_4365_80d0_3d894ce53d19.slice/crio-73cb4557c4bd87771d04e24c1cb1f423e78476273912ee4b2661adb644f94db4 WatchSource:0}: Error finding container 73cb4557c4bd87771d04e24c1cb1f423e78476273912ee4b2661adb644f94db4: Status 404 returned error can't find the container with id 73cb4557c4bd87771d04e24c1cb1f423e78476273912ee4b2661adb644f94db4
Jul 09 18:24:55.972498 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:55.972461    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d379b307-9d56-4287-b10f-f52336003b8d" path="/var/lib/kubelet/pods/d379b307-9d56-4287-b10f-f52336003b8d/volumes"
Jul 09 18:24:56.200583 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:56.200552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd" event={"ID":"d49d7f07-7394-4365-80d0-3d894ce53d19","Type":"ContainerStarted","Data":"0b59a5fc02c7c2e68124da117624be5bcb36139d2f87d67d5df9a4d4af8ea950"}
Jul 09 18:24:56.200740 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:24:56.200612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd" event={"ID":"d49d7f07-7394-4365-80d0-3d894ce53d19","Type":"ContainerStarted","Data":"73cb4557c4bd87771d04e24c1cb1f423e78476273912ee4b2661adb644f94db4"}
Jul 09 18:25:01.222596 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:01.222563    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd_d49d7f07-7394-4365-80d0-3d894ce53d19/storage-initializer/0.log"
Jul 09 18:25:01.222998 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:01.222605    2563 generic.go:356] "Generic (PLEG): container finished" podID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerID="0b59a5fc02c7c2e68124da117624be5bcb36139d2f87d67d5df9a4d4af8ea950" exitCode=1
Jul 09 18:25:01.222998 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:01.222688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd" event={"ID":"d49d7f07-7394-4365-80d0-3d894ce53d19","Type":"ContainerDied","Data":"0b59a5fc02c7c2e68124da117624be5bcb36139d2f87d67d5df9a4d4af8ea950"}
Jul 09 18:25:02.228890 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:02.228857    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd_d49d7f07-7394-4365-80d0-3d894ce53d19/storage-initializer/0.log"
Jul 09 18:25:02.229278 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:02.228922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd" event={"ID":"d49d7f07-7394-4365-80d0-3d894ce53d19","Type":"ContainerStarted","Data":"c120da7e675b8dc4318fc1b25c30629e439f0f8ec69fec5f3ff012e3c77e5daa"}
Jul 09 18:25:04.522291 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:04.522254    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"]
Jul 09 18:25:04.522794 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:04.522564    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd" podUID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerName="storage-initializer" containerID="cri-o://c120da7e675b8dc4318fc1b25c30629e439f0f8ec69fec5f3ff012e3c77e5daa" gracePeriod=30
Jul 09 18:25:06.246511 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.246485    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd_d49d7f07-7394-4365-80d0-3d894ce53d19/storage-initializer/1.log"
Jul 09 18:25:06.246958 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.246940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd_d49d7f07-7394-4365-80d0-3d894ce53d19/storage-initializer/0.log"
Jul 09 18:25:06.247014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.246978    2563 generic.go:356] "Generic (PLEG): container finished" podID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerID="c120da7e675b8dc4318fc1b25c30629e439f0f8ec69fec5f3ff012e3c77e5daa" exitCode=1
Jul 09 18:25:06.247063 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.247020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd" event={"ID":"d49d7f07-7394-4365-80d0-3d894ce53d19","Type":"ContainerDied","Data":"c120da7e675b8dc4318fc1b25c30629e439f0f8ec69fec5f3ff012e3c77e5daa"}
Jul 09 18:25:06.247101 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.247094    2563 scope.go:117] "RemoveContainer" containerID="0b59a5fc02c7c2e68124da117624be5bcb36139d2f87d67d5df9a4d4af8ea950"
Jul 09 18:25:06.379646 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.379603    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd_d49d7f07-7394-4365-80d0-3d894ce53d19/storage-initializer/1.log"
Jul 09 18:25:06.383064 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.383046    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:25:06.440501 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.440462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d7f07-7394-4365-80d0-3d894ce53d19-kserve-provision-location\") pod \"d49d7f07-7394-4365-80d0-3d894ce53d19\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") "
Jul 09 18:25:06.440501 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.440512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqqm6\" (UniqueName: \"kubernetes.io/projected/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-api-access-gqqm6\") pod \"d49d7f07-7394-4365-80d0-3d894ce53d19\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") "
Jul 09 18:25:06.440758 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.440539    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-cabundle-cert\") pod \"d49d7f07-7394-4365-80d0-3d894ce53d19\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") "
Jul 09 18:25:06.440758 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.440555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls\") pod \"d49d7f07-7394-4365-80d0-3d894ce53d19\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") "
Jul 09 18:25:06.440758 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.440576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-rbac-proxy-sar-config\") pod \"d49d7f07-7394-4365-80d0-3d894ce53d19\" (UID: \"d49d7f07-7394-4365-80d0-3d894ce53d19\") "
Jul 09 18:25:06.440917 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.440830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d7f07-7394-4365-80d0-3d894ce53d19-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d49d7f07-7394-4365-80d0-3d894ce53d19" (UID: "d49d7f07-7394-4365-80d0-3d894ce53d19"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:06.440984 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.440935    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d49d7f07-7394-4365-80d0-3d894ce53d19" (UID: "d49d7f07-7394-4365-80d0-3d894ce53d19"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:25:06.441036 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.440985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d49d7f07-7394-4365-80d0-3d894ce53d19" (UID: "d49d7f07-7394-4365-80d0-3d894ce53d19"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:25:06.442752 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.442728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d49d7f07-7394-4365-80d0-3d894ce53d19" (UID: "d49d7f07-7394-4365-80d0-3d894ce53d19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:06.442819 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.442776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-api-access-gqqm6" (OuterVolumeSpecName: "kube-api-access-gqqm6") pod "d49d7f07-7394-4365-80d0-3d894ce53d19" (UID: "d49d7f07-7394-4365-80d0-3d894ce53d19"). InnerVolumeSpecName "kube-api-access-gqqm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:25:06.541364 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.541279    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d7f07-7394-4365-80d0-3d894ce53d19-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:25:06.541364 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.541309    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqm6\" (UniqueName: \"kubernetes.io/projected/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-api-access-gqqm6\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:25:06.541364 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.541320    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:25:06.541364 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.541330    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49d7f07-7394-4365-80d0-3d894ce53d19-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:25:06.541364 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:06.541338    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49d7f07-7394-4365-80d0-3d894ce53d19-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:25:07.251789 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:07.251759    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd_d49d7f07-7394-4365-80d0-3d894ce53d19/storage-initializer/1.log"
Jul 09 18:25:07.252218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:07.251821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd" event={"ID":"d49d7f07-7394-4365-80d0-3d894ce53d19","Type":"ContainerDied","Data":"73cb4557c4bd87771d04e24c1cb1f423e78476273912ee4b2661adb644f94db4"}
Jul 09 18:25:07.252218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:07.251862    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"
Jul 09 18:25:07.252218 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:07.251873    2563 scope.go:117] "RemoveContainer" containerID="c120da7e675b8dc4318fc1b25c30629e439f0f8ec69fec5f3ff012e3c77e5daa"
Jul 09 18:25:07.312264 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:07.312234    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"]
Jul 09 18:25:07.317375 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:07.317352    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4d7d87-predictor-79dfb4685b-lrfjd"]
Jul 09 18:25:07.972287 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:07.972242    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49d7f07-7394-4365-80d0-3d894ce53d19" path="/var/lib/kubelet/pods/d49d7f07-7394-4365-80d0-3d894ce53d19/volumes"
Jul 09 18:25:09.773761 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.773732    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"]
Jul 09 18:25:09.774125 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.774050    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerName="storage-initializer"
Jul 09 18:25:09.774125 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.774065    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerName="storage-initializer"
Jul 09 18:25:09.774125 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.774079    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerName="storage-initializer"
Jul 09 18:25:09.774125 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.774085    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerName="storage-initializer"
Jul 09 18:25:09.774254 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.774134    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerName="storage-initializer"
Jul 09 18:25:09.774254 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.774142    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49d7f07-7394-4365-80d0-3d894ce53d19" containerName="storage-initializer"
Jul 09 18:25:09.778525 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.778500    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.781303 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.781283    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-45121-predictor-serving-cert\""
Jul 09 18:25:09.781446 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.781416    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:25:09.781446 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.781439    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:25:09.781600 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.781290    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8d4cq\""
Jul 09 18:25:09.781600 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.781301    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:25:09.781600 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.781369    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:25:09.781742 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.781290    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-45121-kube-rbac-proxy-sar-config\""
Jul 09 18:25:09.789333 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.789309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"]
Jul 09 18:25:09.867502 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.867470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62bceed5-cdca-4da7-8954-dfd844566c81-proxy-tls\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.867673 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.867510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.867673 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.867532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttbf\" (UniqueName: \"kubernetes.io/projected/62bceed5-cdca-4da7-8954-dfd844566c81-kube-api-access-zttbf\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.867673 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.867548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-cabundle-cert\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.867673 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.867566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bceed5-cdca-4da7-8954-dfd844566c81-kserve-provision-location\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.968356 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.968324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62bceed5-cdca-4da7-8954-dfd844566c81-proxy-tls\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.968540 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.968363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.968540 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.968390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttbf\" (UniqueName: \"kubernetes.io/projected/62bceed5-cdca-4da7-8954-dfd844566c81-kube-api-access-zttbf\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.968540 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.968424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-cabundle-cert\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.968540 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.968443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bceed5-cdca-4da7-8954-dfd844566c81-kserve-provision-location\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.969016 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.968987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bceed5-cdca-4da7-8954-dfd844566c81-kserve-provision-location\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.969210 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.969185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.969284 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.969254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-cabundle-cert\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.971065 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.971042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62bceed5-cdca-4da7-8954-dfd844566c81-proxy-tls\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:09.978517 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:09.978493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttbf\" (UniqueName: \"kubernetes.io/projected/62bceed5-cdca-4da7-8954-dfd844566c81-kube-api-access-zttbf\") pod \"raw-sklearn-45121-predictor-77c554ddcb-rfjm5\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") " pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:10.090546 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:10.090462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:10.243029 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:10.242998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"]
Jul 09 18:25:10.246009 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:25:10.245982    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bceed5_cdca_4da7_8954_dfd844566c81.slice/crio-581371fa5e4a54077ad08b36e35fd9941c7bc55f3e4548a09c46d43ade9af599 WatchSource:0}: Error finding container 581371fa5e4a54077ad08b36e35fd9941c7bc55f3e4548a09c46d43ade9af599: Status 404 returned error can't find the container with id 581371fa5e4a54077ad08b36e35fd9941c7bc55f3e4548a09c46d43ade9af599
Jul 09 18:25:10.265091 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:10.265065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" event={"ID":"62bceed5-cdca-4da7-8954-dfd844566c81","Type":"ContainerStarted","Data":"581371fa5e4a54077ad08b36e35fd9941c7bc55f3e4548a09c46d43ade9af599"}
Jul 09 18:25:11.270615 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:11.270576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" event={"ID":"62bceed5-cdca-4da7-8954-dfd844566c81","Type":"ContainerStarted","Data":"f28aa71fdb47d526ad9bb49a982a2f4acda4d6f00d1fc4dd3101e6f097b9803f"}
Jul 09 18:25:14.285483 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:14.285448    2563 generic.go:356] "Generic (PLEG): container finished" podID="62bceed5-cdca-4da7-8954-dfd844566c81" containerID="f28aa71fdb47d526ad9bb49a982a2f4acda4d6f00d1fc4dd3101e6f097b9803f" exitCode=0
Jul 09 18:25:14.285943 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:14.285526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" event={"ID":"62bceed5-cdca-4da7-8954-dfd844566c81","Type":"ContainerDied","Data":"f28aa71fdb47d526ad9bb49a982a2f4acda4d6f00d1fc4dd3101e6f097b9803f"}
Jul 09 18:25:15.291289 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:15.291256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" event={"ID":"62bceed5-cdca-4da7-8954-dfd844566c81","Type":"ContainerStarted","Data":"568c9182d808277b0d61facb40d1f2709c40ec9cc0bd1c92644a8627080dac33"}
Jul 09 18:25:15.291785 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:15.291318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" event={"ID":"62bceed5-cdca-4da7-8954-dfd844566c81","Type":"ContainerStarted","Data":"548df3525e67c64834d7b04143cf4887a69627f8626121f51ec3b4b67991d716"}
Jul 09 18:25:15.291785 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:15.291659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:15.291785 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:15.291700    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:15.292986 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:15.292960    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:25:15.315910 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:15.315868    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podStartSLOduration=2.831173278 podStartE2EDuration="6.315853831s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484680553s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:15.314592969 +0000 UTC m=+1287.764175407" watchObservedRunningTime="2026-07-09 18:25:15.315853831 +0000 UTC m=+1287.765436254"
Jul 09 18:25:16.295863 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:16.295824    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:25:21.300332 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:21.300301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:25:21.300751 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:21.300724    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:25:31.301680 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:31.301634    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:25:41.301121 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:41.301079    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:25:51.300971 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:25:51.300934    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:26:01.301695 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:01.301648    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:26:11.301316 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:11.301229    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:26:21.301292 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:21.301248    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:26:31.301635 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:31.301601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:26:40.028213 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:40.028166    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"]
Jul 09 18:26:40.028749 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:40.028479    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" containerID="cri-o://548df3525e67c64834d7b04143cf4887a69627f8626121f51ec3b4b67991d716" gracePeriod=30
Jul 09 18:26:40.028749 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:40.028519    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kube-rbac-proxy" containerID="cri-o://568c9182d808277b0d61facb40d1f2709c40ec9cc0bd1c92644a8627080dac33" gracePeriod=30
Jul 09 18:26:40.653560 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:40.653526    2563 generic.go:356] "Generic (PLEG): container finished" podID="62bceed5-cdca-4da7-8954-dfd844566c81" containerID="568c9182d808277b0d61facb40d1f2709c40ec9cc0bd1c92644a8627080dac33" exitCode=2
Jul 09 18:26:40.653751 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:40.653607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" event={"ID":"62bceed5-cdca-4da7-8954-dfd844566c81","Type":"ContainerDied","Data":"568c9182d808277b0d61facb40d1f2709c40ec9cc0bd1c92644a8627080dac33"}
Jul 09 18:26:41.296242 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:41.296204    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 09 18:26:41.301576 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:41.301553    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 09 18:26:44.670208 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.670177    2563 generic.go:356] "Generic (PLEG): container finished" podID="62bceed5-cdca-4da7-8954-dfd844566c81" containerID="548df3525e67c64834d7b04143cf4887a69627f8626121f51ec3b4b67991d716" exitCode=0
Jul 09 18:26:44.670629 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.670248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" event={"ID":"62bceed5-cdca-4da7-8954-dfd844566c81","Type":"ContainerDied","Data":"548df3525e67c64834d7b04143cf4887a69627f8626121f51ec3b4b67991d716"}
Jul 09 18:26:44.686504 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.686479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:26:44.810756 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.810648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zttbf\" (UniqueName: \"kubernetes.io/projected/62bceed5-cdca-4da7-8954-dfd844566c81-kube-api-access-zttbf\") pod \"62bceed5-cdca-4da7-8954-dfd844566c81\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") "
Jul 09 18:26:44.810756 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.810704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-kube-rbac-proxy-sar-config\") pod \"62bceed5-cdca-4da7-8954-dfd844566c81\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") "
Jul 09 18:26:44.810756 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.810729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-cabundle-cert\") pod \"62bceed5-cdca-4da7-8954-dfd844566c81\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") "
Jul 09 18:26:44.811055 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.810763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bceed5-cdca-4da7-8954-dfd844566c81-kserve-provision-location\") pod \"62bceed5-cdca-4da7-8954-dfd844566c81\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") "
Jul 09 18:26:44.811055 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.810803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62bceed5-cdca-4da7-8954-dfd844566c81-proxy-tls\") pod \"62bceed5-cdca-4da7-8954-dfd844566c81\" (UID: \"62bceed5-cdca-4da7-8954-dfd844566c81\") "
Jul 09 18:26:44.811178 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.811146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bceed5-cdca-4da7-8954-dfd844566c81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62bceed5-cdca-4da7-8954-dfd844566c81" (UID: "62bceed5-cdca-4da7-8954-dfd844566c81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:44.811223 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.811173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62bceed5-cdca-4da7-8954-dfd844566c81" (UID: "62bceed5-cdca-4da7-8954-dfd844566c81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:44.811223 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.811160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62bceed5-cdca-4da7-8954-dfd844566c81" (UID: "62bceed5-cdca-4da7-8954-dfd844566c81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:44.812893 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.812865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bceed5-cdca-4da7-8954-dfd844566c81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62bceed5-cdca-4da7-8954-dfd844566c81" (UID: "62bceed5-cdca-4da7-8954-dfd844566c81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:44.812992 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.812922    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bceed5-cdca-4da7-8954-dfd844566c81-kube-api-access-zttbf" (OuterVolumeSpecName: "kube-api-access-zttbf") pod "62bceed5-cdca-4da7-8954-dfd844566c81" (UID: "62bceed5-cdca-4da7-8954-dfd844566c81"). InnerVolumeSpecName "kube-api-access-zttbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:44.911663 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.911625    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bceed5-cdca-4da7-8954-dfd844566c81-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:26:44.911663 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.911656    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62bceed5-cdca-4da7-8954-dfd844566c81-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:26:44.911663 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.911669    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttbf\" (UniqueName: \"kubernetes.io/projected/62bceed5-cdca-4da7-8954-dfd844566c81-kube-api-access-zttbf\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:26:44.911904 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.911679    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:26:44.911904 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:44.911687    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62bceed5-cdca-4da7-8954-dfd844566c81-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:26:45.140515 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140428    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"]
Jul 09 18:26:45.140739 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140726    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container"
Jul 09 18:26:45.140788 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140740    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container"
Jul 09 18:26:45.140788 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140759    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="storage-initializer"
Jul 09 18:26:45.140788 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140765    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="storage-initializer"
Jul 09 18:26:45.140788 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140774    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kube-rbac-proxy"
Jul 09 18:26:45.140788 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140779    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kube-rbac-proxy"
Jul 09 18:26:45.140957 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140829    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kserve-container"
Jul 09 18:26:45.140957 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.140838    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" containerName="kube-rbac-proxy"
Jul 09 18:26:45.144039 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.144019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.146749 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.146716    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-c9334-predictor-serving-cert\""
Jul 09 18:26:45.146982 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.146967    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-c9334-kube-rbac-proxy-sar-config\""
Jul 09 18:26:45.161324 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.161295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"]
Jul 09 18:26:45.316161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.316116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad9c54a-c384-40e4-bcd6-592824bba3ff-proxy-tls\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.316161 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.316164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5st\" (UniqueName: \"kubernetes.io/projected/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-api-access-qn5st\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.316387 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.316195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.316387 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.316231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-cabundle-cert\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.316387 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.316291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kserve-provision-location\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.417438 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.417380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad9c54a-c384-40e4-bcd6-592824bba3ff-proxy-tls\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.417612 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.417450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5st\" (UniqueName: \"kubernetes.io/projected/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-api-access-qn5st\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.417612 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.417475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.417612 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.417499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-cabundle-cert\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.417612 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.417577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kserve-provision-location\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.418308 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.418285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kserve-provision-location\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.418524 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.418505    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.419715 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.419692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-cabundle-cert\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.419955 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.419939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad9c54a-c384-40e4-bcd6-592824bba3ff-proxy-tls\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.428539 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.428519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5st\" (UniqueName: \"kubernetes.io/projected/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-api-access-qn5st\") pod \"raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.455029 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.454998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:45.589666 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.589634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"]
Jul 09 18:26:45.592888 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:26:45.592857    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad9c54a_c384_40e4_bcd6_592824bba3ff.slice/crio-76038d6070153a119fa8a26eb1590eb6151f731bdbb8005ede7e8de410d62dbf WatchSource:0}: Error finding container 76038d6070153a119fa8a26eb1590eb6151f731bdbb8005ede7e8de410d62dbf: Status 404 returned error can't find the container with id 76038d6070153a119fa8a26eb1590eb6151f731bdbb8005ede7e8de410d62dbf
Jul 09 18:26:45.676023 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.675935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" event={"ID":"2ad9c54a-c384-40e4-bcd6-592824bba3ff","Type":"ContainerStarted","Data":"b0ac2fca33c78bfc86671b85d62748958e27ec9040b0e791f996af918512457a"}
Jul 09 18:26:45.676394 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.676032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" event={"ID":"2ad9c54a-c384-40e4-bcd6-592824bba3ff","Type":"ContainerStarted","Data":"76038d6070153a119fa8a26eb1590eb6151f731bdbb8005ede7e8de410d62dbf"}
Jul 09 18:26:45.677860 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.677841    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"
Jul 09 18:26:45.677971 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.677838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5" event={"ID":"62bceed5-cdca-4da7-8954-dfd844566c81","Type":"ContainerDied","Data":"581371fa5e4a54077ad08b36e35fd9941c7bc55f3e4548a09c46d43ade9af599"}
Jul 09 18:26:45.678037 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.677997    2563 scope.go:117] "RemoveContainer" containerID="568c9182d808277b0d61facb40d1f2709c40ec9cc0bd1c92644a8627080dac33"
Jul 09 18:26:45.694487 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.694469    2563 scope.go:117] "RemoveContainer" containerID="548df3525e67c64834d7b04143cf4887a69627f8626121f51ec3b4b67991d716"
Jul 09 18:26:45.702283 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.702264    2563 scope.go:117] "RemoveContainer" containerID="f28aa71fdb47d526ad9bb49a982a2f4acda4d6f00d1fc4dd3101e6f097b9803f"
Jul 09 18:26:45.713849 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.713823    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"]
Jul 09 18:26:45.717862 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.717840    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-45121-predictor-77c554ddcb-rfjm5"]
Jul 09 18:26:45.972209 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:45.972126    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62bceed5-cdca-4da7-8954-dfd844566c81" path="/var/lib/kubelet/pods/62bceed5-cdca-4da7-8954-dfd844566c81/volumes"
Jul 09 18:26:49.701127 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:49.701092    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerID="b0ac2fca33c78bfc86671b85d62748958e27ec9040b0e791f996af918512457a" exitCode=0
Jul 09 18:26:49.701127 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:49.701129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" event={"ID":"2ad9c54a-c384-40e4-bcd6-592824bba3ff","Type":"ContainerDied","Data":"b0ac2fca33c78bfc86671b85d62748958e27ec9040b0e791f996af918512457a"}
Jul 09 18:26:50.707020 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:50.706985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" event={"ID":"2ad9c54a-c384-40e4-bcd6-592824bba3ff","Type":"ContainerStarted","Data":"12cff19374d0b7240ac65315836bb757542455354e2a9cb8450cc8db2bbce67c"}
Jul 09 18:26:50.707375 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:50.707049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" event={"ID":"2ad9c54a-c384-40e4-bcd6-592824bba3ff","Type":"ContainerStarted","Data":"f6c9fd45c4dcfdbb902b846cae61903e91b36e4dbd79c1ba25b5fd7c585e8554"}
Jul 09 18:26:50.707375 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:50.707253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:50.727814 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:50.727769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podStartSLOduration=2.317726039 podStartE2EDuration="5.727754935s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.410028896s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:50.72656974 +0000 UTC m=+1383.176152163" watchObservedRunningTime="2026-07-09 18:26:50.727754935 +0000 UTC m=+1383.177337358"
Jul 09 18:26:51.711369 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:51.711333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:51.712781 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:51.712753    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:26:52.714827 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:52.714787    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:26:57.719598 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:57.719564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:26:57.720153 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:26:57.720125    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:27:07.720372 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:27:07.720329    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:27:17.720958 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:27:17.720917    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:27:27.720235 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:27:27.720188    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:27:37.720972 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:27:37.720930    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:27:47.720981 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:27:47.720886    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:27:57.720397 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:27:57.720356    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:28:07.721034 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:07.721003    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:28:15.226399 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:15.226362    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"]
Jul 09 18:28:15.226863 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:15.226635    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" containerID="cri-o://f6c9fd45c4dcfdbb902b846cae61903e91b36e4dbd79c1ba25b5fd7c585e8554" gracePeriod=30
Jul 09 18:28:15.226863 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:15.226672    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kube-rbac-proxy" containerID="cri-o://12cff19374d0b7240ac65315836bb757542455354e2a9cb8450cc8db2bbce67c" gracePeriod=30
Jul 09 18:28:16.040513 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:16.040481    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerID="12cff19374d0b7240ac65315836bb757542455354e2a9cb8450cc8db2bbce67c" exitCode=2
Jul 09 18:28:16.040694 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:16.040526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" event={"ID":"2ad9c54a-c384-40e4-bcd6-592824bba3ff","Type":"ContainerDied","Data":"12cff19374d0b7240ac65315836bb757542455354e2a9cb8450cc8db2bbce67c"}
Jul 09 18:28:17.715120 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:17.715078    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 09 18:28:17.720873 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:17.720838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 09 18:28:20.059266 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.059235    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerID="f6c9fd45c4dcfdbb902b846cae61903e91b36e4dbd79c1ba25b5fd7c585e8554" exitCode=0
Jul 09 18:28:20.059585 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.059298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" event={"ID":"2ad9c54a-c384-40e4-bcd6-592824bba3ff","Type":"ContainerDied","Data":"f6c9fd45c4dcfdbb902b846cae61903e91b36e4dbd79c1ba25b5fd7c585e8554"}
Jul 09 18:28:20.177658 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.177634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:28:20.303023 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.302944    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad9c54a-c384-40e4-bcd6-592824bba3ff-proxy-tls\") pod \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") "
Jul 09 18:28:20.303023 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.302980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kserve-provision-location\") pod \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") "
Jul 09 18:28:20.303023 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.303019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn5st\" (UniqueName: \"kubernetes.io/projected/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-api-access-qn5st\") pod \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") "
Jul 09 18:28:20.303254 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.303142    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-rbac-proxy-sar-config\") pod \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") "
Jul 09 18:28:20.303254 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.303204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-cabundle-cert\") pod \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\" (UID: \"2ad9c54a-c384-40e4-bcd6-592824bba3ff\") "
Jul 09 18:28:20.303365 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.303342    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ad9c54a-c384-40e4-bcd6-592824bba3ff" (UID: "2ad9c54a-c384-40e4-bcd6-592824bba3ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:20.303548 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.303530    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kserve-provision-location\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:28:20.303632 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.303475    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ad9c54a-c384-40e4-bcd6-592824bba3ff" (UID: "2ad9c54a-c384-40e4-bcd6-592824bba3ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:20.303693 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.303666    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ad9c54a-c384-40e4-bcd6-592824bba3ff" (UID: "2ad9c54a-c384-40e4-bcd6-592824bba3ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:20.305130 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.305102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad9c54a-c384-40e4-bcd6-592824bba3ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ad9c54a-c384-40e4-bcd6-592824bba3ff" (UID: "2ad9c54a-c384-40e4-bcd6-592824bba3ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:20.305247 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.305172    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-api-access-qn5st" (OuterVolumeSpecName: "kube-api-access-qn5st") pod "2ad9c54a-c384-40e4-bcd6-592824bba3ff" (UID: "2ad9c54a-c384-40e4-bcd6-592824bba3ff"). InnerVolumeSpecName "kube-api-access-qn5st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:20.404468 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.404441    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn5st\" (UniqueName: \"kubernetes.io/projected/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-api-access-qn5st\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:28:20.404468 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.404469    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:28:20.404607 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.404480    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ad9c54a-c384-40e4-bcd6-592824bba3ff-cabundle-cert\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:28:20.404607 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:20.404503    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ad9c54a-c384-40e4-bcd6-592824bba3ff-proxy-tls\") on node \"ip-10-0-133-76.ec2.internal\" DevicePath \"\""
Jul 09 18:28:21.065330 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:21.065302    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"
Jul 09 18:28:21.065755 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:21.065333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq" event={"ID":"2ad9c54a-c384-40e4-bcd6-592824bba3ff","Type":"ContainerDied","Data":"76038d6070153a119fa8a26eb1590eb6151f731bdbb8005ede7e8de410d62dbf"}
Jul 09 18:28:21.065755 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:21.065397    2563 scope.go:117] "RemoveContainer" containerID="12cff19374d0b7240ac65315836bb757542455354e2a9cb8450cc8db2bbce67c"
Jul 09 18:28:21.073730 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:21.073714    2563 scope.go:117] "RemoveContainer" containerID="f6c9fd45c4dcfdbb902b846cae61903e91b36e4dbd79c1ba25b5fd7c585e8554"
Jul 09 18:28:21.080798 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:21.080781    2563 scope.go:117] "RemoveContainer" containerID="b0ac2fca33c78bfc86671b85d62748958e27ec9040b0e791f996af918512457a"
Jul 09 18:28:21.090365 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:21.090340    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"]
Jul 09 18:28:21.099257 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:21.099235    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c9334-predictor-695df9fcfb-7vnqq"]
Jul 09 18:28:21.972207 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:21.972162    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" path="/var/lib/kubelet/pods/2ad9c54a-c384-40e4-bcd6-592824bba3ff/volumes"
Jul 09 18:28:47.952349 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:47.952320    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:28:47.954005 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:47.953983    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:28:52.972199 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972169    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5fmck/must-gather-m9jpj"]
Jul 09 18:28:52.974544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972463    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="storage-initializer"
Jul 09 18:28:52.974544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972474    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="storage-initializer"
Jul 09 18:28:52.974544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972484    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kube-rbac-proxy"
Jul 09 18:28:52.974544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972490    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kube-rbac-proxy"
Jul 09 18:28:52.974544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container"
Jul 09 18:28:52.974544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container"
Jul 09 18:28:52.974544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972556    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kube-rbac-proxy"
Jul 09 18:28:52.974544 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.972565    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad9c54a-c384-40e4-bcd6-592824bba3ff" containerName="kserve-container"
Jul 09 18:28:52.975420 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.975393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5fmck/must-gather-m9jpj"
Jul 09 18:28:52.978070 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.978046    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5fmck\"/\"kube-root-ca.crt\""
Jul 09 18:28:52.978191 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.978135    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5fmck\"/\"default-dockercfg-9sz9n\""
Jul 09 18:28:52.978458 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.978443    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5fmck\"/\"openshift-service-ca.crt\""
Jul 09 18:28:52.990325 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:52.990299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5fmck/must-gather-m9jpj"]
Jul 09 18:28:53.043500 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.043474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2c0164db-bbf3-49e8-99e0-e5798454320d-must-gather-output\") pod \"must-gather-m9jpj\" (UID: \"2c0164db-bbf3-49e8-99e0-e5798454320d\") " pod="openshift-must-gather-5fmck/must-gather-m9jpj"
Jul 09 18:28:53.043646 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.043502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87h7\" (UniqueName: \"kubernetes.io/projected/2c0164db-bbf3-49e8-99e0-e5798454320d-kube-api-access-x87h7\") pod \"must-gather-m9jpj\" (UID: \"2c0164db-bbf3-49e8-99e0-e5798454320d\") " pod="openshift-must-gather-5fmck/must-gather-m9jpj"
Jul 09 18:28:53.144876 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.144848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2c0164db-bbf3-49e8-99e0-e5798454320d-must-gather-output\") pod \"must-gather-m9jpj\" (UID: \"2c0164db-bbf3-49e8-99e0-e5798454320d\") " pod="openshift-must-gather-5fmck/must-gather-m9jpj"
Jul 09 18:28:53.144876 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.144875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87h7\" (UniqueName: \"kubernetes.io/projected/2c0164db-bbf3-49e8-99e0-e5798454320d-kube-api-access-x87h7\") pod \"must-gather-m9jpj\" (UID: \"2c0164db-bbf3-49e8-99e0-e5798454320d\") " pod="openshift-must-gather-5fmck/must-gather-m9jpj"
Jul 09 18:28:53.145166 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.145147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2c0164db-bbf3-49e8-99e0-e5798454320d-must-gather-output\") pod \"must-gather-m9jpj\" (UID: \"2c0164db-bbf3-49e8-99e0-e5798454320d\") " pod="openshift-must-gather-5fmck/must-gather-m9jpj"
Jul 09 18:28:53.153795 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.153765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87h7\" (UniqueName: \"kubernetes.io/projected/2c0164db-bbf3-49e8-99e0-e5798454320d-kube-api-access-x87h7\") pod \"must-gather-m9jpj\" (UID: \"2c0164db-bbf3-49e8-99e0-e5798454320d\") " pod="openshift-must-gather-5fmck/must-gather-m9jpj"
Jul 09 18:28:53.284509 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.284421    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5fmck/must-gather-m9jpj"
Jul 09 18:28:53.411530 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.411502    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5fmck/must-gather-m9jpj"]
Jul 09 18:28:53.413856 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:28:53.413829    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c0164db_bbf3_49e8_99e0_e5798454320d.slice/crio-2b6cbd8a9e76714bd9601a4ba69d2e6d29c75de3962b48e9e1307ec40c8703b2 WatchSource:0}: Error finding container 2b6cbd8a9e76714bd9601a4ba69d2e6d29c75de3962b48e9e1307ec40c8703b2: Status 404 returned error can't find the container with id 2b6cbd8a9e76714bd9601a4ba69d2e6d29c75de3962b48e9e1307ec40c8703b2
Jul 09 18:28:53.415671 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:53.415653    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:28:54.203958 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:54.203925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fmck/must-gather-m9jpj" event={"ID":"2c0164db-bbf3-49e8-99e0-e5798454320d","Type":"ContainerStarted","Data":"2b6cbd8a9e76714bd9601a4ba69d2e6d29c75de3962b48e9e1307ec40c8703b2"}
Jul 09 18:28:55.212604 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:55.212563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fmck/must-gather-m9jpj" event={"ID":"2c0164db-bbf3-49e8-99e0-e5798454320d","Type":"ContainerStarted","Data":"8edaa40e3781f4c43ef4c7f8fa12cf3ed3292ea0335572a0aaabf798038590ef"}
Jul 09 18:28:55.213068 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:55.212619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fmck/must-gather-m9jpj" event={"ID":"2c0164db-bbf3-49e8-99e0-e5798454320d","Type":"ContainerStarted","Data":"574f7a5c56f7db31a1f393353c8c82b1c588511ed0ed1b7af6109cad7796cba5"}
Jul 09 18:28:55.232294 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:55.232233    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5fmck/must-gather-m9jpj" podStartSLOduration=2.465089191 podStartE2EDuration="3.232211613s" totalImagesPullingTime="767.122422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:55.229759515 +0000 UTC m=+1507.679341940" watchObservedRunningTime="2026-07-09 18:28:55.232211613 +0000 UTC m=+1507.681794037"
Jul 09 18:28:55.616957 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:55.616879    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j6fjk_50672b24-2d0d-4afc-b7de-ac97da06cb99/global-pull-secret-syncer/0.log"
Jul 09 18:28:55.816014 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:55.815982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9rrpv_dde90adf-88d2-4060-8384-205e8325b6b7/konnectivity-agent/0.log"
Jul 09 18:28:55.889966 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:28:55.889892    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-76.ec2.internal_9722980453de5610e5dce33afc71c5c6/haproxy/0.log"
Jul 09 18:29:00.109763 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:00.109729    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5dxl_6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8/node-exporter/0.log"
Jul 09 18:29:00.139073 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:00.139042    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5dxl_6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8/kube-rbac-proxy/0.log"
Jul 09 18:29:00.169064 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:00.169041    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5dxl_6fab0ae1-f8b1-4c8f-8eb0-ddab2ebb2da8/init-textfile/0.log"
Jul 09 18:29:00.627080 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:00.627048    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-7kxt8_3fc53002-8732-41dd-9dd8-82f73a25d95a/prometheus-operator-admission-webhook/0.log"
Jul 09 18:29:02.298578 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:02.298551    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-6zkf5_185ed8c1-2274-4cef-b764-caa9d400a138/networking-console-plugin/0.log"
Jul 09 18:29:03.456388 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.456361    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8675cfc849-56w68_7e95a149-6c15-4bb1-97f1-e7f44faef832/console/0.log"
Jul 09 18:29:03.494526 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.494470    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-jnsfg_7189db45-a477-45a9-85a3-a6af87915e9f/download-server/0.log"
Jul 09 18:29:03.674122 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.674076    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"]
Jul 09 18:29:03.679710 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.679677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.692833 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.692810    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"]
Jul 09 18:29:03.738380 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.738292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-lib-modules\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.738380 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.738354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-proc\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.738622 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.738479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-sys\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.738622 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.738537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-podres\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.738622 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.738577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gffmx\" (UniqueName: \"kubernetes.io/projected/f4fc2ff7-0998-4412-82f9-b65b6544025f-kube-api-access-gffmx\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839053 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-lib-modules\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-proc\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-sys\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-podres\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-lib-modules\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffmx\" (UniqueName: \"kubernetes.io/projected/f4fc2ff7-0998-4412-82f9-b65b6544025f-kube-api-access-gffmx\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839239 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-proc\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839563 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-sys\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.839563 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.839300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4fc2ff7-0998-4412-82f9-b65b6544025f-podres\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.849542 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.849509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffmx\" (UniqueName: \"kubernetes.io/projected/f4fc2ff7-0998-4412-82f9-b65b6544025f-kube-api-access-gffmx\") pod \"perf-node-gather-daemonset-rt6hx\" (UID: \"f4fc2ff7-0998-4412-82f9-b65b6544025f\") " pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:03.989866 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:03.989786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:04.170750 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:04.170721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"]
Jul 09 18:29:04.173615 ip-10-0-133-76 kubenswrapper[2563]: W0709 18:29:04.173582    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf4fc2ff7_0998_4412_82f9_b65b6544025f.slice/crio-d9bb7bfea07eb73333e210f796c667f2bcb963f3665d2a6985e6aa8239d9abb2 WatchSource:0}: Error finding container d9bb7bfea07eb73333e210f796c667f2bcb963f3665d2a6985e6aa8239d9abb2: Status 404 returned error can't find the container with id d9bb7bfea07eb73333e210f796c667f2bcb963f3665d2a6985e6aa8239d9abb2
Jul 09 18:29:04.265365 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:04.263713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx" event={"ID":"f4fc2ff7-0998-4412-82f9-b65b6544025f","Type":"ContainerStarted","Data":"819ce4f86c65d83593c350edf11ed3d72508c9986c3099d5e081dabb9798239c"}
Jul 09 18:29:04.265365 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:04.263777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx" event={"ID":"f4fc2ff7-0998-4412-82f9-b65b6544025f","Type":"ContainerStarted","Data":"d9bb7bfea07eb73333e210f796c667f2bcb963f3665d2a6985e6aa8239d9abb2"}
Jul 09 18:29:04.265365 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:04.264490    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:04.289346 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:04.289299    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx" podStartSLOduration=1.289281668 podStartE2EDuration="1.289281668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:04.288663307 +0000 UTC m=+1516.738245731" watchObservedRunningTime="2026-07-09 18:29:04.289281668 +0000 UTC m=+1516.738864092"
Jul 09 18:29:05.036608 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:05.036575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r2bq9_111b9162-7a38-462e-bbcf-eade884b4331/dns/0.log"
Jul 09 18:29:05.058204 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:05.058165    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r2bq9_111b9162-7a38-462e-bbcf-eade884b4331/kube-rbac-proxy/0.log"
Jul 09 18:29:05.183425 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:05.183383    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rdjmn_20a6402d-7d33-4e8b-875a-02afdf64090b/dns-node-resolver/0.log"
Jul 09 18:29:05.673467 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:05.673440    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59bdd44495-fgpfm_04bd49d0-3b4d-462a-a7b6-3dc7cc9ce03f/registry/0.log"
Jul 09 18:29:05.719467 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:05.719440    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-msphn_5c353489-1b68-489f-ace9-2e463f42d74b/node-ca/0.log"
Jul 09 18:29:06.990516 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:06.990486    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-psr2d_111b8c89-cb2b-4293-8264-1a2307cb57ed/serve-healthcheck-canary/0.log"
Jul 09 18:29:07.625869 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:07.625840    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kss8w_01c2b861-d2e0-459f-889b-fad5bf3abaea/kube-rbac-proxy/0.log"
Jul 09 18:29:07.650987 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:07.650963    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kss8w_01c2b861-d2e0-459f-889b-fad5bf3abaea/exporter/0.log"
Jul 09 18:29:07.675528 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:07.675506    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kss8w_01c2b861-d2e0-459f-889b-fad5bf3abaea/extractor/0.log"
Jul 09 18:29:09.874584 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:09.874512    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-c2jqn_ab481449-e45a-41fa-8edd-025edab5691e/server/0.log"
Jul 09 18:29:10.048977 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:10.048949    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-c2nkw_a787447b-6580-4fbd-8ffd-a179e13cb70d/seaweedfs/0.log"
Jul 09 18:29:11.287903 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:11.287872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5fmck/perf-node-gather-daemonset-rt6hx"
Jul 09 18:29:14.254437 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:14.254398    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-g4nd5_d10396de-6fa9-4cd3-a6d4-95615a49dce8/migrator/0.log"
Jul 09 18:29:14.276846 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:14.276817    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-g4nd5_d10396de-6fa9-4cd3-a6d4-95615a49dce8/graceful-termination/0.log"
Jul 09 18:29:15.990924 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:15.990897    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zl75c_937aa4bf-82d0-4e87-8755-3b8e87055e9f/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:29:16.012595 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.012570    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zl75c_937aa4bf-82d0-4e87-8755-3b8e87055e9f/egress-router-binary-copy/0.log"
Jul 09 18:29:16.035608 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.035581    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zl75c_937aa4bf-82d0-4e87-8755-3b8e87055e9f/cni-plugins/0.log"
Jul 09 18:29:16.061048 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.061027    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zl75c_937aa4bf-82d0-4e87-8755-3b8e87055e9f/bond-cni-plugin/0.log"
Jul 09 18:29:16.082783 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.082759    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zl75c_937aa4bf-82d0-4e87-8755-3b8e87055e9f/routeoverride-cni/0.log"
Jul 09 18:29:16.104665 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.104644    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zl75c_937aa4bf-82d0-4e87-8755-3b8e87055e9f/whereabouts-cni-bincopy/0.log"
Jul 09 18:29:16.126978 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.126958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zl75c_937aa4bf-82d0-4e87-8755-3b8e87055e9f/whereabouts-cni/0.log"
Jul 09 18:29:16.165761 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.165740    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gvpv7_ac8f5fb4-6756-4ddf-9e3d-a60cdcc86a38/kube-multus/0.log"
Jul 09 18:29:16.313147 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.313084    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p55kt_9434adf9-7671-4b69-a142-438262daa264/network-metrics-daemon/0.log"
Jul 09 18:29:16.335273 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:16.335250    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p55kt_9434adf9-7671-4b69-a142-438262daa264/kube-rbac-proxy/0.log"
Jul 09 18:29:17.413593 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.413564    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-controller/0.log"
Jul 09 18:29:17.434489 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.434459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/1.log"
Jul 09 18:29:17.434667 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.434528    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovn-acl-logging/0.log"
Jul 09 18:29:17.455246 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.455211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/kube-rbac-proxy-node/0.log"
Jul 09 18:29:17.476343 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.476320    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:29:17.498066 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.498043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/northd/0.log"
Jul 09 18:29:17.522174 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.522152    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/nbdb/0.log"
Jul 09 18:29:17.546833 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.546805    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/sbdb/0.log"
Jul 09 18:29:17.594271 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:17.594243    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5dg8_5696ada7-3869-49f9-b1fb-c7ad87198205/ovnkube-controller/0.log"
Jul 09 18:29:18.951930 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:18.951898    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-h8zvk_75696b63-fe4e-4141-981f-81204168ea86/check-endpoints/0.log"
Jul 09 18:29:19.010461 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:19.010432    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ns4p4_c5affad4-9ffa-40cb-8a15-ff9c327859a0/network-check-target-container/0.log"
Jul 09 18:29:20.144001 ip-10-0-133-76 kubenswrapper[2563]: I0709 18:29:20.143960    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-cpmn4_70d4383f-8db0-40d3-80cb-2a614fbcd8b0/iptables-alerter/0.log"