Sep 17 17:32:25.008654 ip-10-0-133-107 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:32:25.008669 ip-10-0-133-107 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:32:25.008678 ip-10-0-133-107 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:32:25.008879 ip-10-0-133-107 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:32:36.214914 ip-10-0-133-107 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:32:36.214930 ip-10-0-133-107 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 00a6f02849984f809a1926052bb2de9b --
Sep 17 17:34:43.260012 ip-10-0-133-107 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:34:46.650390 ip-10-0-133-107 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:46.650390 ip-10-0-133-107 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:34:46.650390 ip-10-0-133-107 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:46.650390 ip-10-0-133-107 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:34:46.650390 ip-10-0-133-107 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:46.651358 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.650457    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:34:46.653231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653214    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:46.653231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653230    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653234    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653238    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653241    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653244    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653246    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653249    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653251    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653254    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653256    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653259    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653261    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653265    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653267    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653270    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653273    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653275    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653278    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653281    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653283    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:46.653291 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653286    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653289    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653291    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653294    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653296    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653299    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653302    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653305    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653307    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653310    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653313    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653315    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653318    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653320    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653324    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653328    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653331    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653333    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653336    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:46.653809 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653339    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653343    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653346    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653349    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653352    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653355    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653358    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653361    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653363    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653366    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653369    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653372    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653374    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653377    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653380    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653384    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653387    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653389    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653392    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653395    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:46.654265 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653397    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653400    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653402    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653405    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653407    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653410    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653412    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653415    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653419    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653423    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653426    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653429    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653431    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653434    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653436    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653439    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653442    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653444    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653447    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:46.654766 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653450    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653453    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653456    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653460    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653463    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653465    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653468    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653470    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653473    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653476    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653478    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653481    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653483    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653486    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653488    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653491    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653493    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653496    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653499    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.653501    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:46.655235 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654209    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654217    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654220    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654223    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654226    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654229    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654232    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654235    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654238    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654240    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654243    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654246    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654249    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654252    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654255    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654258    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654260    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654266    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654270    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:46.655729 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654274    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654278    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654280    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654283    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654286    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654289    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654292    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654294    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654297    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654300    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654302    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654305    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654307    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654310    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654313    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654315    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654318    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654321    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654323    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654326    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:46.656237 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654328    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654331    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654333    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654338    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654342    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654345    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654348    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654351    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654354    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654357    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654360    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654363    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654365    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654368    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654371    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654373    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654376    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654378    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654381    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654384    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:46.656748 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654386    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654389    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654392    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654394    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654397    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654399    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654402    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654404    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654407    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654409    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654412    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654415    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654417    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654419    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654422    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654424    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654427    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654429    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654432    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654435    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:46.657231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654439    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654441    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654444    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654447    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654452    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654455    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654458    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654461    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654464    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654467    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654470    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654473    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654476    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654478    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654481    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654484    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654486    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654490    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654492    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:46.657718 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654495    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654860    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654865    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654868    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654871    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654874    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654877    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654880    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654882    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654885    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654888    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654890    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654893    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654897    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654900    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654904    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654906    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654909    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654912    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654914    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:46.658179 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654917    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654920    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654922    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654925    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654928    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654930    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654933    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654935    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654938    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654941    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654944    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654946    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654950    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654954    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654958    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654961    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654963    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654966    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654969    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654971    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:46.658673 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654974    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654976    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654979    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654981    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654985    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654989    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654992    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654995    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.654998    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655001    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655004    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655006    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655009    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655011    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655014    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655017    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655019    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655022    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655024    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:46.659163 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655027    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655029    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655032    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655034    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655037    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655039    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655042    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655044    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655047    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655049    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655051    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655054    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655057    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655060    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655062    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655065    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655067    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655070    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655073    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655076    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:46.659639 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655081    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655083    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655086    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655088    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655091    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655093    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655096    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655098    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655101    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655104    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655106    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655109    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655111    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655114    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655116    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655119    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655121    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655124    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655128    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655131    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:46.660156 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655134    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655201    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655241    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655251    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655255    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655260    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655263    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655268    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655272    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655277    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655281    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655292    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655298    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655303    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655306    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655309    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655312    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655315    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655318    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655322    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655325    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655328    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655331    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655335    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:34:46.660635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655338    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655341    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655345    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655348    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655351    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655354    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655357    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655361    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655364    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655368    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655371    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655374    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655377    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655380    2581 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655382    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655387    2581 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655390    2581 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655395    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655398    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655401    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655406    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655409    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655412    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655415    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655418    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:34:46.661229 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655421    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655424    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655427    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655430    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655433    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655436    2581 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655440    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655443    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655446    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655449    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655452    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655455    2581 flags.go:64] FLAG: --help="false"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655459    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-133-107.ec2.internal"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655462    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655465    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655468    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655471    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655475    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655477    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655480    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655483    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655486    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655489    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655492    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:34:46.661901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655496    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655499    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655502    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655507    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655510    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655513    2581 flags.go:64] FLAG: --lock-file=""
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655516    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655520    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655523    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655529    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655532    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655535    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655538    2581 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655541    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655544    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655547    2581 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655550    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655554    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655557    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655561    2581 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655564    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655567    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655570    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655573    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655576    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:34:46.662651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655579    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655582    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655589    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655592    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655595    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655599    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655602    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655607    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655612    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655615    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655619    2581 flags.go:64] FLAG: --port="10250"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655623    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655626    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0730a4725dbca7ebd"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655629    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655632    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655636    2581 flags.go:64] FLAG: --register-node="true"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655639    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655642    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655645    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655648    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655651    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655655    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655658    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655661    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655664    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:34:46.663692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655667    2581 flags.go:64] FLAG: --runonce="false"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655670    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655673    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655676    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655679    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655682    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655685    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655688    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655692    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655695    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655698    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655716    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655720    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655725    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655729    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655733    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655739    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655743    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655746    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655750    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655753    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655756    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655760    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655763    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655766    2581 flags.go:64] FLAG: --v="2"
Sep 17 17:34:46.664573 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655771    2581 flags.go:64] FLAG: --version="false"
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655774    2581 flags.go:64] FLAG: --vmodule=""
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655778    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.655781    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655913    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655919    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655922    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655925    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655928    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655931    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655934    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655936    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655939    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655941    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655944    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655947    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655949    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655952    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655954    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655957    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:46.665218 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655959    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655962    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655966    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655968    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655973    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655976    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655978    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655981    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655984    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655987    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655990    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655993    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655996    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.655998    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656001    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656004    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656006    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656009    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656012    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656015    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:46.665731 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656017    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656020    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656022    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656025    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656028    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656030    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656033    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656035    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656038    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656041    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656044    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656046    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656049    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656052    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656058    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656061    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656065    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656069    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656072    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:46.666299 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656075    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656079    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656082    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656084    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656088    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656090    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656093    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656095    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656098    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656101    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656104    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656106    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656109    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656111    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656114    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656116    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656119    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656122    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656124    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:46.666789 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656127    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656129    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656132    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656135    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656137    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656140    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656143    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656146    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656150    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656152    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656156    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656159    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656162    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656164    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656167    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656170    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656173    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656175    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656178    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656195    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:46.667260 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656199    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656202    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656205    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656208    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.656211    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.656805    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.665484    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.665501    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.665522    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.665527    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665584    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665588    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665591    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665595    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665597    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:46.667779 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665601    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665603    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665606    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665609    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665611    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665614    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665617    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665620    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665622    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665625    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665628    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665630    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665633    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665637    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665642    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665645    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665648    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665651    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665654    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:46.668154 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665657    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665660    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665664    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665668    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665671    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665675    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665678    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665682    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665685    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665688    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665690    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665693    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665697    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665700    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665716    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665719    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665722    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665725    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665728    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665731    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:46.668624 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665734    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665737    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665741    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665744    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665748    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665750    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665753    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665756    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665758    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665762    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665766    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665769    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665772    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665776    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665779    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665781    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665784    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665788    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665791    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665793    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:46.669147 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665796    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665799    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665802    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665804    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665807    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665810    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665812    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665815    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665817    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665820    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665822    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665825    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665827    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665830    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665833    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665836    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665838    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665841    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665844    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:46.669669 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665846    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665849    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665852    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665854    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665857    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665859    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665862    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665865    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665868    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665870    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665873    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665876    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665878    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665881    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665884    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:46.670212 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.665886    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.665891    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666023    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666029    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666032    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666035    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666038    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666041    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666044    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666046    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666049    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666052    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666054    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666057    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666061    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:46.670581 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666063    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666066    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666069    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666071    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666074    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666077    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666079    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666082    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666084    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666087    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666090    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666092    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666095    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666098    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666100    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666103    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666105    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666108    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666111    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666114    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:46.670960 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666116    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666119    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666122    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666124    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666127    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666131    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666136    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666139    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666141    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666144    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666146    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666149    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666152    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666154    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666157    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666160    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666162    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666165    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666168    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666170    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:46.671467 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666173    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666176    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666178    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666182    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666186    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666189    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666192    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666195    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666198    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666201    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666205    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666207    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666210    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666212    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666215    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666217    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666220    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666222    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666225    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:46.671959 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666228    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666230    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666233    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666235    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666238    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666241    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666245    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666248    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666251    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666254    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666257    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666260    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666263    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666266    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666269    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666271    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666274    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666277    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666279    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:46.672417 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666282    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666284    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666287    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666290    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666292    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666295    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666297    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:46.666300    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.666305    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.666416    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.668989    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.669834    2581 server.go:1019] "Starting client certificate rotation"
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.669932    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:46.672961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.669962    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:46.673281 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.672095    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:46.673281 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.673193    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:46.685186 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.685165    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:34:46.686790 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.686773    2581 log.go:25] "Validated CRI v1 image API"
Sep 17 17:34:46.687585 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.687569    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:34:46.691276 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.691258    2581 fs.go:135] Filesystem UUIDs: map[68469760-836c-436d-a49c-1e2ceea27cda:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bf56729c-ab52-4fe0-afb0-0a406d9bd149:/dev/nvme0n1p3]
Sep 17 17:34:46.691354 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.691277    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:34:46.696279 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.696169    2581 manager.go:216] Machine: {Timestamp:2026-09-17 17:34:46.694832977 +0000 UTC m=+1.926863787 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098094 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c6a5a767ae62b32655cb9d30d34d0 SystemUUID:ec2c6a5a-767a-e62b-3265-5cb9d30d34d0 BootID:00a6f028-4998-4f80-9a19-26052bb2de9b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:96:83:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:96:83:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:1a:51:f0:48:5c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:34:46.696279 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.696275    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:34:46.696390 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.696349    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:34:46.698159 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.698119    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:34:46.698582 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.698348    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:34:46.700661 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.700648    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:34:46.700726 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.700665    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:34:46.700726 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.700673    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:46.700726 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.700692    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:46.701867 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.701852    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:34:46.703858 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.703840    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:46.706122 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706109    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:46.706238 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706229    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:34:46.706273 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706258    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:34:46.706273 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706270    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:34:46.706327 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706293    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:34:46.706327 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706303    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:34:46.706327 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706311    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:34:46.706646 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706635    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:46.706682 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.706654    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:46.708583 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.708566    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:34:46.710586 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.710573    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:34:46.710626 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.710603    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:34:46.711742 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711730    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711747    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711754    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711759    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711768    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711774    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711789    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711794    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711801    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:34:46.711809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711806    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:34:46.712050 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711814    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:34:46.712050 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711820    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:34:46.712050 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711844    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:34:46.712050 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.711850    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.715326    2581 server.go:1268] "Started kubelet"
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.715395    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.715412    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.715468    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.716600    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.717291    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.717416    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.717662    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:34:46.719516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.718732    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:34:46.716076 ip-10-0-133-107 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:34:46.720377 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.719771    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:34:46.724589 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.724573    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:34:46.724589 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.724583    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:46.724821 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.724684    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:46.725094 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.725075    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:34:46.725094 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.725078    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:34:46.725218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.725111    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:34:46.725218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.725197    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:34:46.725218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.725206    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:34:46.725959 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.725944    2581 factory.go:55] Registering systemd factory
Sep 17 17:34:46.726032 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.725963    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:34:46.726240 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.726226    2581 factory.go:153] Registering CRI-O factory
Sep 17 17:34:46.726240 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.726241    2581 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:34:46.726383 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.726306    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:34:46.726383 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.726323    2581 factory.go:103] Registering Raw factory
Sep 17 17:34:46.726383 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.726338    2581 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:34:46.726716 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.726690    2581 manager.go:349] Starting recovery of all containers
Sep 17 17:34:46.758338 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.758306    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:34:46.758480 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.758433    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:34:46.758619 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.725278    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-107.ec2.internal.18d62c30071622c3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-107.ec2.internal,UID:ip-10-0-133-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-107.ec2.internal,},FirstTimestamp:2026-09-17 17:34:46.715294403 +0000 UTC m=+1.947325202,LastTimestamp:2026-09-17 17:34:46.715294403 +0000 UTC m=+1.947325202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-107.ec2.internal,}"
Sep 17 17:34:46.761393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.761366    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tx8l"
Sep 17 17:34:46.764398 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.764381    2581 manager.go:354] Recovery completed
Sep 17 17:34:46.768576 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.768559    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:46.770429 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.770409    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:46.770510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.770443    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:46.770510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.770458    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:46.770984 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.770971    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:34:46.770984 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.770983    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:34:46.771071 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.770998    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:46.771071 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.770983    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:46.771713 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.771687    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:46.771775 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.771724    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:46.771775 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.771734    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:46.772688 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.772629    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-107.ec2.internal.18d62c300a5f60f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-107.ec2.internal,UID:ip-10-0-133-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-107.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-107.ec2.internal,},FirstTimestamp:2026-09-17 17:34:46.770426097 +0000 UTC m=+2.002456932,LastTimestamp:2026-09-17 17:34:46.770426097 +0000 UTC m=+2.002456932,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-107.ec2.internal,}"
Sep 17 17:34:46.772965 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.772952    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tx8l"
Sep 17 17:34:46.774224 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.774212    2581 policy_none.go:49] "None policy: Start"
Sep 17 17:34:46.774259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.774229    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:34:46.774259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.774240    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:34:46.776144 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.776135    2581 policy_none.go:47] "Start"
Sep 17 17:34:46.809500 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.809353    2581 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.809525    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.809539    2581 server.go:87] "Starting device plugin registration server"
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.809767    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.809785    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.809907    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.810019    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.810029    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.810491    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:34:46.830135 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.810530    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:46.890909 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.890873    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:34:46.892092 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.892059    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:34:46.892201 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.892113    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:34:46.892201 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.892137    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:34:46.892201 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.892182    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:34:46.894683 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.894666    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:46.910789 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.910733    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:46.911587 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.911572    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:46.911673 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.911604    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:46.911673 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.911620    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:46.911673 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.911651    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-107.ec2.internal"
Sep 17 17:34:46.918847 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.918833    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-107.ec2.internal"
Sep 17 17:34:46.918896 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.918856    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-107.ec2.internal\": node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:46.938971 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:46.938953    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:46.992548 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.992520    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"]
Sep 17 17:34:46.992649 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.992608    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:46.993521 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.993506    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:46.993599 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.993532    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:46.993599 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.993542    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:46.995780 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.995767    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:46.995901 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.995885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:46.995948 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.995912    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:46.996484 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.996469    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:46.996560 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.996495    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:46.996560 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.996506    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:46.996560 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.996472    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:46.996694 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.996574    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:46.996694 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.996591    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:46.998778 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.998763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:46.998877 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.998789    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:46.999435 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.999418    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:46.999515 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.999441    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:46.999515 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:46.999451    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:47.021229 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.021208    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-107.ec2.internal\" not found" node="ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.025496 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.025481    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-107.ec2.internal\" not found" node="ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.026607 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.026592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3351252bd9bc869b867a2349bd56cda0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"3351252bd9bc869b867a2349bd56cda0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.026653 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.026615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3351252bd9bc869b867a2349bd56cda0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"3351252bd9bc869b867a2349bd56cda0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.026653 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.026630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0a53354102c1933fab32ef2c66f3050-config\") pod \"kube-apiserver-proxy-ip-10-0-133-107.ec2.internal\" (UID: \"a0a53354102c1933fab32ef2c66f3050\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.040021 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.040004    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.127614 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.127584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3351252bd9bc869b867a2349bd56cda0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"3351252bd9bc869b867a2349bd56cda0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.127614 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.127614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0a53354102c1933fab32ef2c66f3050-config\") pod \"kube-apiserver-proxy-ip-10-0-133-107.ec2.internal\" (UID: \"a0a53354102c1933fab32ef2c66f3050\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.127806 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.127634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3351252bd9bc869b867a2349bd56cda0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"3351252bd9bc869b867a2349bd56cda0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.127806 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.127670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0a53354102c1933fab32ef2c66f3050-config\") pod \"kube-apiserver-proxy-ip-10-0-133-107.ec2.internal\" (UID: \"a0a53354102c1933fab32ef2c66f3050\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.127806 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.127696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3351252bd9bc869b867a2349bd56cda0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"3351252bd9bc869b867a2349bd56cda0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.127806 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.127670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3351252bd9bc869b867a2349bd56cda0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"3351252bd9bc869b867a2349bd56cda0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.140834 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.140801    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.241550 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.241480    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.324714 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.324684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.330198 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.329840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:47.342557 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.342537    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.443078 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.443040    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.543562 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.543500    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.643978 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.643954    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.670136 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.670110    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:34:47.670503 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.670242    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:47.744536 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.744512    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.778949 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.778891    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:29:46 +0000 UTC" deadline="2028-04-07 07:30:21.780267426 +0000 UTC"
Sep 17 17:34:47.778949 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.778950    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13621h55m34.001324506s"
Sep 17 17:34:47.828971 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.828948    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:47.844601 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.844566    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.932681 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.932646    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:47.945637 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:47.945609    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
Sep 17 17:34:47.967258 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:47.967229    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a53354102c1933fab32ef2c66f3050.slice/crio-af6f48e7ad65cc0b43578d8437c37fa2b12cd0eb65f42a4379644f34cdc76ee0 WatchSource:0}: Error finding container af6f48e7ad65cc0b43578d8437c37fa2b12cd0eb65f42a4379644f34cdc76ee0: Status 404 returned error can't find the container with id af6f48e7ad65cc0b43578d8437c37fa2b12cd0eb65f42a4379644f34cdc76ee0
Sep 17 17:34:47.967902 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:47.967881    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3351252bd9bc869b867a2349bd56cda0.slice/crio-300a14f32a3c6e794f1df1956367efa25b4e01981ff16dd86efcf359c770df5a WatchSource:0}: Error finding container 300a14f32a3c6e794f1df1956367efa25b4e01981ff16dd86efcf359c770df5a: Status 404 returned error can't find the container with id 300a14f32a3c6e794f1df1956367efa25b4e01981ff16dd86efcf359c770df5a
Sep 17 17:34:47.975039 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.975021    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:47.975390 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:47.975369    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:48.025844 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.025809    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:48.045311 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.045291    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:48.046201 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.046189    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
Sep 17 17:34:48.062234 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.062210    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:48.668442 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.668361    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:48.708473 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.708221    2581 apiserver.go:52] "Watching apiserver"
Sep 17 17:34:48.718001 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.717973    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:34:48.718526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.718491    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-ch8qr","kube-system/global-pull-secret-syncer-2bl7r","kube-system/konnectivity-agent-bfjrr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp","openshift-image-registry/node-ca-jjc62","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal","openshift-multus/multus-additional-cni-plugins-krjlf","openshift-network-diagnostics/network-check-target-5225q","openshift-network-operator/iptables-alerter-kl9k2","kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal","openshift-cluster-node-tuning-operator/tuned-q62bf","openshift-dns/node-resolver-mbsv8","openshift-multus/multus-jbqnx","openshift-multus/network-metrics-daemon-kc6h8"]
Sep 17 17:34:48.721043 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.721020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.723150 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.723130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:48.723253 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.723208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:34:48.724248 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.724225    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:48.724340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.724257    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:48.724854 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.724798    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x75ps\""
Sep 17 17:34:48.725609 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.725339    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:48.725609 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.725458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.728602 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.727939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.730274 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.730252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.732502 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.732445    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.734743 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.734725    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:48.734833 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.734786    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:34:48.735748 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-socket-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.735841 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-modprobe-d\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.735841 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-system-cni-dir\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.735841 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-cnibin\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.735995 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-device-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.735995 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-sys\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.735995 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-var-lib-kubelet\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.735995 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-tuned\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.735995 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.735988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sxp\" (UniqueName: \"kubernetes.io/projected/91affe17-634c-4ac8-b1a7-b60485e21db4-kube-api-access-f8sxp\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-sys-fs\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmg6\" (UniqueName: \"kubernetes.io/projected/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-kube-api-access-8rmg6\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysctl-d\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-run\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysconfig\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-systemd\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849w7\" (UniqueName: \"kubernetes.io/projected/ad9b1c7f-425a-4e7c-a433-5400163131a1-kube-api-access-849w7\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-registration-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysctl-conf\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-lib-modules\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-kubernetes\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-host\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad9b1c7f-425a-4e7c-a433-5400163131a1-tmp\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcqg\" (UniqueName: \"kubernetes.io/projected/b665814a-b3dd-4590-9dbf-63559f1f5681-kube-api-access-zpcqg\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.736641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736565    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:34:48.737162 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.737162 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736752    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gf4j8\""
Sep 17 17:34:48.737162 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.736760    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:34:48.737162 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737041    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:34:48.737343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737213    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:34:48.737343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737247    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:34:48.737465 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737451    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:34:48.737525 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737500    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:34:48.737640 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737624    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:34:48.737735 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737680    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:34:48.737941 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737898    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lfwbm\""
Sep 17 17:34:48.737941 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.737907    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5cldl\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738174    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738238    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738353    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738441    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738581    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738687    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9zv5\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738789    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738879    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.738986    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:34:48.739329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.739010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.739854 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.739373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:34:48.744963 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.744935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.747259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.746932    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:34:48.747259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.746974    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2zc68\""
Sep 17 17:34:48.747259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.747071    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:34:48.747259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.747101    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-54bsf\""
Sep 17 17:34:48.747259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.747157    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:48.747259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.747208    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:48.747590 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.747293    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:34:48.748844 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.748822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.749523 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.749505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:34:48.749891 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.749873    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:34:48.750034 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.750017    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlgpr\""
Sep 17 17:34:48.750875 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.750777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.750875 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.750848    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:34:48.752074 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.751743    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gv8ch\""
Sep 17 17:34:48.752074 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.751953    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:34:48.786400 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.786372    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:48.826206 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.826176    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:34:48.837358 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-kubernetes\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.837526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:48.837526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:48.837526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-netns\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.837526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-run-netns\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.837526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-host\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.837526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad9b1c7f-425a-4e7c-a433-5400163131a1-tmp\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.837526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcqg\" (UniqueName: \"kubernetes.io/projected/b665814a-b3dd-4590-9dbf-63559f1f5681-kube-api-access-zpcqg\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-system-cni-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-cni-binary-copy\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-cni-bin\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-system-cni-dir\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-systemd-units\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z5q7\" (UniqueName: \"kubernetes.io/projected/fb65f3ad-31a3-4b27-b383-8859526e81d4-kube-api-access-7z5q7\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86tj\" (UniqueName: \"kubernetes.io/projected/f2887851-3bcf-41ff-95cd-5182504f880b-kube-api-access-s86tj\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-sys\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-var-lib-kubelet\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-daemon-config\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-kubelet\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.837904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-var-lib-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-cni-bin\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.837975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovn-node-metrics-cert\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sxp\" (UniqueName: \"kubernetes.io/projected/91affe17-634c-4ac8-b1a7-b60485e21db4-kube-api-access-f8sxp\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-sys-fs\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-var-lib-kubelet\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszfh\" (UniqueName: \"kubernetes.io/projected/753a888b-2139-4ac2-afbf-2085bf6df478-kube-api-access-lszfh\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-sys\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-system-cni-dir\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6d832701-c936-4fba-b54d-df1ed6594c63-kubelet-config\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysctl-d\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fc71244-a845-45e1-b684-12cf8dd012f6-agent-certs\") pod \"konnectivity-agent-bfjrr\" (UID: \"0fc71244-a845-45e1-b684-12cf8dd012f6\") " pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-cnibin\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-conf-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-host\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.838506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-log-socket\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysconfig\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838387    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-sys-fs\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-host\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-kubernetes\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.838775    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysctl-d\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.838838    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs podName:b665814a-b3dd-4590-9dbf-63559f1f5681 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:49.33881704 +0000 UTC m=+4.570847836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs") pod "network-metrics-daemon-kc6h8" (UID: "b665814a-b3dd-4590-9dbf-63559f1f5681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysconfig\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.838392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849w7\" (UniqueName: \"kubernetes.io/projected/ad9b1c7f-425a-4e7c-a433-5400163131a1-kube-api-access-849w7\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-registration-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-multus-certs\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-etc-kubernetes\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovnkube-config\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2887851-3bcf-41ff-95cd-5182504f880b-iptables-alerter-script\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.839416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-lib-modules\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-slash\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-systemd\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-env-overrides\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6d832701-c936-4fba-b54d-df1ed6594c63-dbus\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fc71244-a845-45e1-b684-12cf8dd012f6-konnectivity-ca\") pod \"konnectivity-agent-bfjrr\" (UID: \"0fc71244-a845-45e1-b684-12cf8dd012f6\") " pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2887851-3bcf-41ff-95cd-5182504f880b-host-slash\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-kubelet\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-socket-dir-parent\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-socket-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-modprobe-d\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-cnibin\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-device-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-cni-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.840226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-ovn\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-tuned\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-hostroot\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-lib-modules\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wzc8\" (UniqueName: \"kubernetes.io/projected/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-kube-api-access-6wzc8\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrdq\" (UniqueName: \"kubernetes.io/projected/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-kube-api-access-qrrdq\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmg6\" (UniqueName: \"kubernetes.io/projected/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-kube-api-access-8rmg6\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-serviceca\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/753a888b-2139-4ac2-afbf-2085bf6df478-hosts-file\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-run\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.839977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-cni-multus\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-etc-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.841004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-node-log\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-cni-netd\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-systemd\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovnkube-script-lib\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753a888b-2139-4ac2-afbf-2085bf6df478-tmp-dir\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysctl-conf\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-k8s-cni-cncf-io\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-socket-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-modprobe-d\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-registration-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91affe17-634c-4ac8-b1a7-b60485e21db4-cnibin\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-device-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-systemd\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.841783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-run\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.842594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.840906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-sysctl-conf\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.842594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.841226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91affe17-634c-4ac8-b1a7-b60485e21db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.842594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.842312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad9b1c7f-425a-4e7c-a433-5400163131a1-tmp\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.844084 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.844051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ad9b1c7f-425a-4e7c-a433-5400163131a1-etc-tuned\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.851844 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.851823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-849w7\" (UniqueName: \"kubernetes.io/projected/ad9b1c7f-425a-4e7c-a433-5400163131a1-kube-api-access-849w7\") pod \"tuned-q62bf\" (UID: \"ad9b1c7f-425a-4e7c-a433-5400163131a1\") " pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:48.852293 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.852271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcqg\" (UniqueName: \"kubernetes.io/projected/b665814a-b3dd-4590-9dbf-63559f1f5681-kube-api-access-zpcqg\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:48.852954 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.852924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sxp\" (UniqueName: \"kubernetes.io/projected/91affe17-634c-4ac8-b1a7-b60485e21db4-kube-api-access-f8sxp\") pod \"multus-additional-cni-plugins-krjlf\" (UID: \"91affe17-634c-4ac8-b1a7-b60485e21db4\") " pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:48.854069 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.854048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmg6\" (UniqueName: \"kubernetes.io/projected/afd3ba62-6b4f-4a4f-adf9-4a3191c398ca-kube-api-access-8rmg6\") pod \"aws-ebs-csi-driver-node-2nkcp\" (UID: \"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:48.855502 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.855481    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nn7vz"
Sep 17 17:34:48.864950 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.864931    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nn7vz"
Sep 17 17:34:48.897590 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.897538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal" event={"ID":"a0a53354102c1933fab32ef2c66f3050","Type":"ContainerStarted","Data":"af6f48e7ad65cc0b43578d8437c37fa2b12cd0eb65f42a4379644f34cdc76ee0"}
Sep 17 17:34:48.898808 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.898775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal" event={"ID":"3351252bd9bc869b867a2349bd56cda0","Type":"ContainerStarted","Data":"300a14f32a3c6e794f1df1956367efa25b4e01981ff16dd86efcf359c770df5a"}
Sep 17 17:34:48.941235 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-socket-dir-parent\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941235 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-cni-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941235 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-ovn\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.941235 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-hostroot\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941235 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-socket-dir-parent\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941235 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wzc8\" (UniqueName: \"kubernetes.io/projected/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-kube-api-access-6wzc8\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrdq\" (UniqueName: \"kubernetes.io/projected/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-kube-api-access-qrrdq\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-serviceca\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/753a888b-2139-4ac2-afbf-2085bf6df478-hosts-file\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-cni-multus\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-cni-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-etc-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-node-log\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.941441    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-ovn\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-cni-netd\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-hostroot\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.941495    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret podName:6d832701-c936-4fba-b54d-df1ed6594c63 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:49.44147736 +0000 UTC m=+4.673508173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret") pod "global-pull-secret-syncer-2bl7r" (UID: "6d832701-c936-4fba-b54d-df1ed6594c63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-etc-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/753a888b-2139-4ac2-afbf-2085bf6df478-hosts-file\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-node-log\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-cni-multus\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.941655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-cni-netd\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovnkube-script-lib\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753a888b-2139-4ac2-afbf-2085bf6df478-tmp-dir\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-k8s-cni-cncf-io\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-netns\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-run-netns\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-system-cni-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-serviceca\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753a888b-2139-4ac2-afbf-2085bf6df478-tmp-dir\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-k8s-cni-cncf-io\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-run-netns\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovnkube-script-lib\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-netns\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-system-cni-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.941862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-cni-binary-copy\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.942537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-cni-bin\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-systemd-units\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z5q7\" (UniqueName: \"kubernetes.io/projected/fb65f3ad-31a3-4b27-b383-8859526e81d4-kube-api-access-7z5q7\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86tj\" (UniqueName: \"kubernetes.io/projected/f2887851-3bcf-41ff-95cd-5182504f880b-kube-api-access-s86tj\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-daemon-config\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-systemd-units\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-kubelet\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-cni-bin\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-var-lib-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-cni-bin\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovn-node-metrics-cert\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszfh\" (UniqueName: \"kubernetes.io/projected/753a888b-2139-4ac2-afbf-2085bf6df478-kube-api-access-lszfh\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6d832701-c936-4fba-b54d-df1ed6594c63-kubelet-config\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fc71244-a845-45e1-b684-12cf8dd012f6-agent-certs\") pod \"konnectivity-agent-bfjrr\" (UID: \"0fc71244-a845-45e1-b684-12cf8dd012f6\") " pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-cnibin\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-conf-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-host\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.943343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-log-socket\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-multus-certs\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-cni-binary-copy\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-etc-kubernetes\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-etc-kubernetes\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovnkube-config\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2887851-3bcf-41ff-95cd-5182504f880b-iptables-alerter-script\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-slash\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-systemd\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-env-overrides\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6d832701-c936-4fba-b54d-df1ed6594c63-dbus\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.942989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fc71244-a845-45e1-b684-12cf8dd012f6-konnectivity-ca\") pod \"konnectivity-agent-bfjrr\" (UID: \"0fc71244-a845-45e1-b684-12cf8dd012f6\") " pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-var-lib-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2887851-3bcf-41ff-95cd-5182504f880b-host-slash\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-kubelet\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6d832701-c936-4fba-b54d-df1ed6594c63-kubelet-config\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-conf-dir\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-openvswitch\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-cni-bin\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-run-systemd\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-slash\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-var-lib-kubelet\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-multus-daemon-config\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovnkube-config\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2887851-3bcf-41ff-95cd-5182504f880b-iptables-alerter-script\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6d832701-c936-4fba-b54d-df1ed6594c63-dbus\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2887851-3bcf-41ff-95cd-5182504f880b-host-slash\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-cnibin\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-log-socket\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb65f3ad-31a3-4b27-b383-8859526e81d4-host-kubelet\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-host-run-multus-certs\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.944967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.943881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-host\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.945760 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.945685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb65f3ad-31a3-4b27-b383-8859526e81d4-ovn-node-metrics-cert\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.945760 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.945695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb65f3ad-31a3-4b27-b383-8859526e81d4-env-overrides\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.945760 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.945749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fc71244-a845-45e1-b684-12cf8dd012f6-konnectivity-ca\") pod \"konnectivity-agent-bfjrr\" (UID: \"0fc71244-a845-45e1-b684-12cf8dd012f6\") " pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:34:48.948954 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.948910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fc71244-a845-45e1-b684-12cf8dd012f6-agent-certs\") pod \"konnectivity-agent-bfjrr\" (UID: \"0fc71244-a845-45e1-b684-12cf8dd012f6\") " pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:34:48.956770 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.956742    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrdq\" (UniqueName: \"kubernetes.io/projected/fa4430ae-bc53-43aa-bc0e-77cb787e0d32-kube-api-access-qrrdq\") pod \"node-ca-jjc62\" (UID: \"fa4430ae-bc53-43aa-bc0e-77cb787e0d32\") " pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:48.956864 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.956823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wzc8\" (UniqueName: \"kubernetes.io/projected/546c6e44-f2b9-45e6-bdb9-6682f3f061d9-kube-api-access-6wzc8\") pod \"multus-jbqnx\" (UID: \"546c6e44-f2b9-45e6-bdb9-6682f3f061d9\") " pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:48.956951 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.956933    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:48.957001 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.956956    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:48.957001 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.956969    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j9w6b for pod openshift-network-diagnostics/network-check-target-5225q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:48.957106 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:48.957027    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b podName:db4b0ca9-0163-42ec-be91-e396a0cdaf9f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:49.457009395 +0000 UTC m=+4.689040196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j9w6b" (UniqueName: "kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b") pod "network-check-target-5225q" (UID: "db4b0ca9-0163-42ec-be91-e396a0cdaf9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:48.959230 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.959195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86tj\" (UniqueName: \"kubernetes.io/projected/f2887851-3bcf-41ff-95cd-5182504f880b-kube-api-access-s86tj\") pod \"iptables-alerter-kl9k2\" (UID: \"f2887851-3bcf-41ff-95cd-5182504f880b\") " pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:48.959588 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.959563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z5q7\" (UniqueName: \"kubernetes.io/projected/fb65f3ad-31a3-4b27-b383-8859526e81d4-kube-api-access-7z5q7\") pod \"ovnkube-node-ch8qr\" (UID: \"fb65f3ad-31a3-4b27-b383-8859526e81d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:48.960010 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:48.959963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszfh\" (UniqueName: \"kubernetes.io/projected/753a888b-2139-4ac2-afbf-2085bf6df478-kube-api-access-lszfh\") pod \"node-resolver-mbsv8\" (UID: \"753a888b-2139-4ac2-afbf-2085bf6df478\") " pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:49.033304 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.033270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q62bf"
Sep 17 17:34:49.043300 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.043256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-krjlf"
Sep 17 17:34:49.058159 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.058131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp"
Sep 17 17:34:49.062000 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.061983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjc62"
Sep 17 17:34:49.070728 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.070694    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:34:49.079506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.079481    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kl9k2"
Sep 17 17:34:49.096230 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.096208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:34:49.102946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.102925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbsv8"
Sep 17 17:34:49.109540 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.109523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbqnx"
Sep 17 17:34:49.345810 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.345721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:49.345964 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.345900    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:49.346023 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.345981    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs podName:b665814a-b3dd-4590-9dbf-63559f1f5681 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.34596093 +0000 UTC m=+5.577991733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs") pod "network-metrics-daemon-kc6h8" (UID: "b665814a-b3dd-4590-9dbf-63559f1f5681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:49.446459 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.446415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:49.446630 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.446583    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:49.446730 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.446661    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret podName:6d832701-c936-4fba-b54d-df1ed6594c63 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.446639153 +0000 UTC m=+5.678669963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret") pod "global-pull-secret-syncer-2bl7r" (UID: "6d832701-c936-4fba-b54d-df1ed6594c63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:49.546994 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.546954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:49.547167 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.547100    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:49.547167 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.547122    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:49.547167 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.547134    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j9w6b for pod openshift-network-diagnostics/network-check-target-5225q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:49.547270 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.547190    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b podName:db4b0ca9-0163-42ec-be91-e396a0cdaf9f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.547174569 +0000 UTC m=+5.779205366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9w6b" (UniqueName: "kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b") pod "network-check-target-5225q" (UID: "db4b0ca9-0163-42ec-be91-e396a0cdaf9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:49.867517 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.867461    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:48 +0000 UTC" deadline="2028-02-14 02:22:35.037931983 +0000 UTC"
Sep 17 17:34:49.867517 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.867509    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12344h47m45.170426796s"
Sep 17 17:34:49.874635 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.874608    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9b1c7f_425a_4e7c_a433_5400163131a1.slice/crio-92a413cc6de9ba5b5adb714744c5afb9d21a057ec4f86bf7dff231f918f4d7c5 WatchSource:0}: Error finding container 92a413cc6de9ba5b5adb714744c5afb9d21a057ec4f86bf7dff231f918f4d7c5: Status 404 returned error can't find the container with id 92a413cc6de9ba5b5adb714744c5afb9d21a057ec4f86bf7dff231f918f4d7c5
Sep 17 17:34:49.875806 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.875777    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4430ae_bc53_43aa_bc0e_77cb787e0d32.slice/crio-e9e120d2e087ff5f50cc2217f944eaf0c78bd42c049144a47ef1f5419894945a WatchSource:0}: Error finding container e9e120d2e087ff5f50cc2217f944eaf0c78bd42c049144a47ef1f5419894945a: Status 404 returned error can't find the container with id e9e120d2e087ff5f50cc2217f944eaf0c78bd42c049144a47ef1f5419894945a
Sep 17 17:34:49.882217 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.882182    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd3ba62_6b4f_4a4f_adf9_4a3191c398ca.slice/crio-78166445338c438f5f8837f22073546d577989b2560d5980208e4ff2a0066c4e WatchSource:0}: Error finding container 78166445338c438f5f8837f22073546d577989b2560d5980208e4ff2a0066c4e: Status 404 returned error can't find the container with id 78166445338c438f5f8837f22073546d577989b2560d5980208e4ff2a0066c4e
Sep 17 17:34:49.883593 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.883572    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb65f3ad_31a3_4b27_b383_8859526e81d4.slice/crio-a9f4777474a78b4afb7e3d30cd03bd0095a9d7cbe4bf030740906045c712e91e WatchSource:0}: Error finding container a9f4777474a78b4afb7e3d30cd03bd0095a9d7cbe4bf030740906045c712e91e: Status 404 returned error can't find the container with id a9f4777474a78b4afb7e3d30cd03bd0095a9d7cbe4bf030740906045c712e91e
Sep 17 17:34:49.884940 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.884915    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91affe17_634c_4ac8_b1a7_b60485e21db4.slice/crio-b6971036cd95781e3734b37bc42c39de1da2cb6536e0f123069c4736e5980103 WatchSource:0}: Error finding container b6971036cd95781e3734b37bc42c39de1da2cb6536e0f123069c4736e5980103: Status 404 returned error can't find the container with id b6971036cd95781e3734b37bc42c39de1da2cb6536e0f123069c4736e5980103
Sep 17 17:34:49.885762 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.885742    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546c6e44_f2b9_45e6_bdb9_6682f3f061d9.slice/crio-a20f56ba6222f70637c42c16be65bb406369f900afbffdd57c4a4201b5c101b6 WatchSource:0}: Error finding container a20f56ba6222f70637c42c16be65bb406369f900afbffdd57c4a4201b5c101b6: Status 404 returned error can't find the container with id a20f56ba6222f70637c42c16be65bb406369f900afbffdd57c4a4201b5c101b6
Sep 17 17:34:49.886919 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.886875    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753a888b_2139_4ac2_afbf_2085bf6df478.slice/crio-b338f221b611b680238844b68dc0bd3c9bf271817c54979eb552666f36dafc67 WatchSource:0}: Error finding container b338f221b611b680238844b68dc0bd3c9bf271817c54979eb552666f36dafc67: Status 404 returned error can't find the container with id b338f221b611b680238844b68dc0bd3c9bf271817c54979eb552666f36dafc67
Sep 17 17:34:49.888656 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.888600    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2887851_3bcf_41ff_95cd_5182504f880b.slice/crio-59b86d94d9c7114f5754e54fd898311aa7fb023ca843f3b557dd7fd5b75140ae WatchSource:0}: Error finding container 59b86d94d9c7114f5754e54fd898311aa7fb023ca843f3b557dd7fd5b75140ae: Status 404 returned error can't find the container with id 59b86d94d9c7114f5754e54fd898311aa7fb023ca843f3b557dd7fd5b75140ae
Sep 17 17:34:49.890320 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:34:49.890243    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc71244_a845_45e1_b684_12cf8dd012f6.slice/crio-3c76284b5e1a059fc866f2dd13308b49c442d82b748b19088006d6c8f4f8fbb8 WatchSource:0}: Error finding container 3c76284b5e1a059fc866f2dd13308b49c442d82b748b19088006d6c8f4f8fbb8: Status 404 returned error can't find the container with id 3c76284b5e1a059fc866f2dd13308b49c442d82b748b19088006d6c8f4f8fbb8
Sep 17 17:34:49.892339 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.892320    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:49.892450 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:49.892423    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:34:49.901156 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.901135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbsv8" event={"ID":"753a888b-2139-4ac2-afbf-2085bf6df478","Type":"ContainerStarted","Data":"b338f221b611b680238844b68dc0bd3c9bf271817c54979eb552666f36dafc67"}
Sep 17 17:34:49.902635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.902609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbqnx" event={"ID":"546c6e44-f2b9-45e6-bdb9-6682f3f061d9","Type":"ContainerStarted","Data":"a20f56ba6222f70637c42c16be65bb406369f900afbffdd57c4a4201b5c101b6"}
Sep 17 17:34:49.903885 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.903865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krjlf" event={"ID":"91affe17-634c-4ac8-b1a7-b60485e21db4","Type":"ContainerStarted","Data":"b6971036cd95781e3734b37bc42c39de1da2cb6536e0f123069c4736e5980103"}
Sep 17 17:34:49.904920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.904899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bfjrr" event={"ID":"0fc71244-a845-45e1-b684-12cf8dd012f6","Type":"ContainerStarted","Data":"3c76284b5e1a059fc866f2dd13308b49c442d82b748b19088006d6c8f4f8fbb8"}
Sep 17 17:34:49.906260 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.906242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"a9f4777474a78b4afb7e3d30cd03bd0095a9d7cbe4bf030740906045c712e91e"}
Sep 17 17:34:49.907281 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.907262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp" event={"ID":"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca","Type":"ContainerStarted","Data":"78166445338c438f5f8837f22073546d577989b2560d5980208e4ff2a0066c4e"}
Sep 17 17:34:49.908279 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.908256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjc62" event={"ID":"fa4430ae-bc53-43aa-bc0e-77cb787e0d32","Type":"ContainerStarted","Data":"e9e120d2e087ff5f50cc2217f944eaf0c78bd42c049144a47ef1f5419894945a"}
Sep 17 17:34:49.909287 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.909267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q62bf" event={"ID":"ad9b1c7f-425a-4e7c-a433-5400163131a1","Type":"ContainerStarted","Data":"92a413cc6de9ba5b5adb714744c5afb9d21a057ec4f86bf7dff231f918f4d7c5"}
Sep 17 17:34:49.910314 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:49.910296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kl9k2" event={"ID":"f2887851-3bcf-41ff-95cd-5182504f880b","Type":"ContainerStarted","Data":"59b86d94d9c7114f5754e54fd898311aa7fb023ca843f3b557dd7fd5b75140ae"}
Sep 17 17:34:50.354572 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.354335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:50.354815 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.354765    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:50.354918 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.354834    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs podName:b665814a-b3dd-4590-9dbf-63559f1f5681 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:52.354815574 +0000 UTC m=+7.586846374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs") pod "network-metrics-daemon-kc6h8" (UID: "b665814a-b3dd-4590-9dbf-63559f1f5681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:50.455258 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.455165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:50.455482 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.455358    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:50.455482 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.455423    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret podName:6d832701-c936-4fba-b54d-df1ed6594c63 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:52.455405622 +0000 UTC m=+7.687436432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret") pod "global-pull-secret-syncer-2bl7r" (UID: "6d832701-c936-4fba-b54d-df1ed6594c63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:50.557953 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.556253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:50.557953 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.556441    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:50.557953 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.556461    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:50.557953 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.556474    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j9w6b for pod openshift-network-diagnostics/network-check-target-5225q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:50.557953 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.556534    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b podName:db4b0ca9-0163-42ec-be91-e396a0cdaf9f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:52.556515755 +0000 UTC m=+7.788546568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9w6b" (UniqueName: "kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b") pod "network-check-target-5225q" (UID: "db4b0ca9-0163-42ec-be91-e396a0cdaf9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:50.867756 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.867631    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:48 +0000 UTC" deadline="2028-05-16 01:22:44.613027491 +0000 UTC"
Sep 17 17:34:50.867756 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.867711    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14551h47m53.745332497s"
Sep 17 17:34:50.893474 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.893442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:50.893621 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.893584    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:34:50.893876 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.893742    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:50.893876 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:50.893831    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:34:50.934263 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.934226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal" event={"ID":"a0a53354102c1933fab32ef2c66f3050","Type":"ContainerStarted","Data":"a8d8a20a13ef8d81a31a0594be30d3baa4f0920d51b7301fe8e66d352d709fa2"}
Sep 17 17:34:50.948299 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.947817    2581 generic.go:356] "Generic (PLEG): container finished" podID="3351252bd9bc869b867a2349bd56cda0" containerID="8e7bd5e1afb03da0861502e7ea8db81a70d7e64e7c91f57df459e598713dc7ad" exitCode=0
Sep 17 17:34:50.948299 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.947876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal" event={"ID":"3351252bd9bc869b867a2349bd56cda0","Type":"ContainerDied","Data":"8e7bd5e1afb03da0861502e7ea8db81a70d7e64e7c91f57df459e598713dc7ad"}
Sep 17 17:34:50.970440 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:50.970281    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal" podStartSLOduration=2.970258143 podStartE2EDuration="2.970258143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:50.952556313 +0000 UTC m=+6.184587131" watchObservedRunningTime="2026-09-17 17:34:50.970258143 +0000 UTC m=+6.202288964"
Sep 17 17:34:51.892614 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:51.892486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:51.893167 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:51.892622    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:34:51.971369 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:51.971333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal" event={"ID":"3351252bd9bc869b867a2349bd56cda0","Type":"ContainerStarted","Data":"2914b80744057b21fe5d7a91df6447390a82d5e980b08c747c65109ae432f8e1"}
Sep 17 17:34:51.996429 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:51.996374    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal" podStartSLOduration=3.996351947 podStartE2EDuration="3.996351947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:51.995368249 +0000 UTC m=+7.227399068" watchObservedRunningTime="2026-09-17 17:34:51.996351947 +0000 UTC m=+7.228382767"
Sep 17 17:34:52.373008 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:52.372357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:52.373008 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.372531    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:52.373008 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.372592    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs podName:b665814a-b3dd-4590-9dbf-63559f1f5681 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.372573813 +0000 UTC m=+11.604604613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs") pod "network-metrics-daemon-kc6h8" (UID: "b665814a-b3dd-4590-9dbf-63559f1f5681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:52.473728 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:52.473425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:52.473728 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.473562    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:52.473728 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.473627    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret podName:6d832701-c936-4fba-b54d-df1ed6594c63 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.473609079 +0000 UTC m=+11.705639891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret") pod "global-pull-secret-syncer-2bl7r" (UID: "6d832701-c936-4fba-b54d-df1ed6594c63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:52.573891 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:52.573851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:52.574061 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.574025    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:52.574061 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.574052    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:52.574178 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.574066    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j9w6b for pod openshift-network-diagnostics/network-check-target-5225q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:52.574178 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.574126    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b podName:db4b0ca9-0163-42ec-be91-e396a0cdaf9f nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.574108433 +0000 UTC m=+11.806139244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9w6b" (UniqueName: "kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b") pod "network-check-target-5225q" (UID: "db4b0ca9-0163-42ec-be91-e396a0cdaf9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:52.893099 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:52.893072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:52.893567 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.893188    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:34:52.895805 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:52.893035    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:52.895927 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:52.895906    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:34:53.892853 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:53.892822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:53.893035 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:53.892957    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:34:54.892544 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:54.892508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:54.893005 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:54.892653    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:34:54.893005 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:54.892740    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:54.893005 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:54.892850    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:34:55.892884 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:55.892853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:55.893344 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:55.892969    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:34:56.405066 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:56.405012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:56.405245 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.405135    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:56.405245 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.405216    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs podName:b665814a-b3dd-4590-9dbf-63559f1f5681 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:04.40519645 +0000 UTC m=+19.637227250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs") pod "network-metrics-daemon-kc6h8" (UID: "b665814a-b3dd-4590-9dbf-63559f1f5681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:56.506000 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:56.505962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:56.506185 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.506098    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:56.506185 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.506163    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret podName:6d832701-c936-4fba-b54d-df1ed6594c63 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:04.506143713 +0000 UTC m=+19.738174551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret") pod "global-pull-secret-syncer-2bl7r" (UID: "6d832701-c936-4fba-b54d-df1ed6594c63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:56.606899 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:56.606860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:56.607069 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.607052    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:56.607136 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.607078    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:56.607136 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.607092    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j9w6b for pod openshift-network-diagnostics/network-check-target-5225q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:56.607269 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.607163    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b podName:db4b0ca9-0163-42ec-be91-e396a0cdaf9f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:04.607143563 +0000 UTC m=+19.839174363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9w6b" (UniqueName: "kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b") pod "network-check-target-5225q" (UID: "db4b0ca9-0163-42ec-be91-e396a0cdaf9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:56.894649 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:56.894141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:56.894649 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.894267    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:34:56.894649 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:56.894457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:56.894649 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:56.894543    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:34:57.892856 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:57.892821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:57.893019 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:57.892954    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:34:58.892844 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:58.892804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:34:58.893337 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:58.892957    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:34:58.893337 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:58.893059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:34:58.893337 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:58.893165    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:34:59.893359 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:34:59.893323    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:34:59.893818 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:34:59.893446    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:00.892404 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:00.892356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:00.892605 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:00.892475    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:00.892605 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:00.892556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:00.892736 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:00.892649    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:01.892946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:01.892904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:01.893402 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:01.893030    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:02.893114 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:02.893077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:02.893536 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:02.893077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:02.893536 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:02.893212    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:02.893536 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:02.893305    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:03.893186 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:03.893106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:03.893566 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:03.893220    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:04.464610 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:04.464573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:04.464801 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.464758    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:04.464878 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.464838    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs podName:b665814a-b3dd-4590-9dbf-63559f1f5681 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:20.46481697 +0000 UTC m=+35.696847786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs") pod "network-metrics-daemon-kc6h8" (UID: "b665814a-b3dd-4590-9dbf-63559f1f5681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:04.565899 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:04.565868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:04.566099 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.566025    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:04.566099 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.566099    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret podName:6d832701-c936-4fba-b54d-df1ed6594c63 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:20.566080954 +0000 UTC m=+35.798111756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret") pod "global-pull-secret-syncer-2bl7r" (UID: "6d832701-c936-4fba-b54d-df1ed6594c63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:04.666417 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:04.666379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:04.666578 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.666550    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:04.666578 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.666573    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:04.666676 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.666583    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j9w6b for pod openshift-network-diagnostics/network-check-target-5225q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:04.666676 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.666634    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b podName:db4b0ca9-0163-42ec-be91-e396a0cdaf9f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:20.666619445 +0000 UTC m=+35.898650246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9w6b" (UniqueName: "kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b") pod "network-check-target-5225q" (UID: "db4b0ca9-0163-42ec-be91-e396a0cdaf9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:04.893196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:04.893109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:04.893617 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:04.893109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:04.893617 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.893231    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:04.893617 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:04.893295    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:05.892609 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:05.892575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:05.892805 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:05.892673    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:06.894098 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:06.894048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:06.894607 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:06.894176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:06.894607 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:06.894192    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:06.894607 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:06.894305    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:07.892511 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:07.892303    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:07.892645 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:07.892621    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:08.004105 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.004071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbsv8" event={"ID":"753a888b-2139-4ac2-afbf-2085bf6df478","Type":"ContainerStarted","Data":"8fd19915bddb4e98e7c10e122668dcfe7a2c53a9f863d5d7a155affda5be73a0"}
Sep 17 17:35:08.005621 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.005598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbqnx" event={"ID":"546c6e44-f2b9-45e6-bdb9-6682f3f061d9","Type":"ContainerStarted","Data":"94b4e3ff77cf8176389184ef6fc2be8d895c5a45c79d2ea7010409b8d9ee3829"}
Sep 17 17:35:08.007575 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.007549    2581 generic.go:356] "Generic (PLEG): container finished" podID="91affe17-634c-4ac8-b1a7-b60485e21db4" containerID="94f083cab4325d75e639abfc53b215a1ec121134c67fb04adf4f226d2289af0f" exitCode=0
Sep 17 17:35:08.007672 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.007627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krjlf" event={"ID":"91affe17-634c-4ac8-b1a7-b60485e21db4","Type":"ContainerDied","Data":"94f083cab4325d75e639abfc53b215a1ec121134c67fb04adf4f226d2289af0f"}
Sep 17 17:35:08.009482 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.009457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bfjrr" event={"ID":"0fc71244-a845-45e1-b684-12cf8dd012f6","Type":"ContainerStarted","Data":"ba134bf99cca1f297dbf62929ec089fb7948c967f4aacccd590b3d5849419ef2"}
Sep 17 17:35:08.013851 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.013425    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:35:08.013952 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.013935    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb65f3ad-31a3-4b27-b383-8859526e81d4" containerID="eaa45c6cf57bfa6d36e2ca0e23fe9dbc22fa047dd33f4518d51762c98fa35372" exitCode=1
Sep 17 17:35:08.014036 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.014015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"fae9347777615956b404d950b8335c97a57f823ca2bbace234678599430f6795"}
Sep 17 17:35:08.014103 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.014091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"8a253bad41a8f0f1446eded038fb88bf50c89cf0c6f50d8aafa3adfe7dcc1da8"}
Sep 17 17:35:08.014152 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.014142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"3c2dc57d9bd11ed8dc923e11db6debc0bf92e8045c670f83126134a4ae314f31"}
Sep 17 17:35:08.014207 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.014193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerDied","Data":"eaa45c6cf57bfa6d36e2ca0e23fe9dbc22fa047dd33f4518d51762c98fa35372"}
Sep 17 17:35:08.014265 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.014241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"38b8f381907b6a3fe08c3440ca089f958dbe3c8951bb47fb16f952a10c26016d"}
Sep 17 17:35:08.015976 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.015939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp" event={"ID":"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca","Type":"ContainerStarted","Data":"b7a1d67b0593891ee9ed81cf0e44d430b59d9b6d741b10fffb8b3743cb1f3971"}
Sep 17 17:35:08.017546 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.017525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjc62" event={"ID":"fa4430ae-bc53-43aa-bc0e-77cb787e0d32","Type":"ContainerStarted","Data":"7d69cff2490b4790b234d58424bb450f36d8a3e49a73d3d563986c8d1eeb5904"}
Sep 17 17:35:08.018972 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.018932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q62bf" event={"ID":"ad9b1c7f-425a-4e7c-a433-5400163131a1","Type":"ContainerStarted","Data":"9fd5115cfddaf2b8eb054dc10a75f6d1fb4f36471a8e8ae01523593c001c58e0"}
Sep 17 17:35:08.027012 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.026972    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mbsv8" podStartSLOduration=3.805234286 podStartE2EDuration="21.026956754s" totalImagesPullingTime="17.221722468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:08.026506108 +0000 UTC m=+23.258536927" watchObservedRunningTime="2026-09-17 17:35:08.026956754 +0000 UTC m=+23.258987574"
Sep 17 17:35:08.047855 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.046071    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bfjrr" podStartSLOduration=3.939032909 podStartE2EDuration="21.046050379s" totalImagesPullingTime="17.10701747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:08.044855532 +0000 UTC m=+23.276886342" watchObservedRunningTime="2026-09-17 17:35:08.046050379 +0000 UTC m=+23.278081251"
Sep 17 17:35:08.088276 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.088206    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jbqnx" podStartSLOduration=3.844043115 podStartE2EDuration="21.088190576s" totalImagesPullingTime="17.244147461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:08.088185641 +0000 UTC m=+23.320216460" watchObservedRunningTime="2026-09-17 17:35:08.088190576 +0000 UTC m=+23.320221396"
Sep 17 17:35:08.089018 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.088962    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q62bf" podStartSLOduration=4.857507876 podStartE2EDuration="22.088947867s" totalImagesPullingTime="17.231439991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:08.066345923 +0000 UTC m=+23.298376743" watchObservedRunningTime="2026-09-17 17:35:08.088947867 +0000 UTC m=+23.320978689"
Sep 17 17:35:08.134233 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.134184    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jjc62" podStartSLOduration=3.905307868 podStartE2EDuration="21.134163853s" totalImagesPullingTime="17.228855985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:08.133936648 +0000 UTC m=+23.365967469" watchObservedRunningTime="2026-09-17 17:35:08.134163853 +0000 UTC m=+23.366194671"
Sep 17 17:35:08.754385 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.754354    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:35:08.824382 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.824285    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:35:08.754378323Z","UUID":"825b1d8e-2743-4833-8402-6b65c1327038","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:35:08.825866 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.825842    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:35:08.825992 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.825878    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:35:08.893369 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.893297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:08.893506 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:08.893431    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:08.893506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:08.893466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:08.893604 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:08.893535    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:09.022917 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:09.022883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kl9k2" event={"ID":"f2887851-3bcf-41ff-95cd-5182504f880b","Type":"ContainerStarted","Data":"e6f420283b9f53c73b107b66a7fdd9c473080e134a62d486c30b91b70e197424"}
Sep 17 17:35:09.026346 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:09.026318    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:35:09.026808 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:09.026734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"679194e3b21621c1e8048293fe55d76206ff1b16d8e49accf944da1f57328702"}
Sep 17 17:35:09.028813 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:09.028768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp" event={"ID":"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca","Type":"ContainerStarted","Data":"8a38323d62b05960550a21aa7255f091e78a6d024bb25525f7599d2227aef3b8"}
Sep 17 17:35:09.042691 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:09.042651    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kl9k2" podStartSLOduration=4.8154389779999995 podStartE2EDuration="22.042634496s" totalImagesPullingTime="17.227195518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:09.042197308 +0000 UTC m=+24.274228127" watchObservedRunningTime="2026-09-17 17:35:09.042634496 +0000 UTC m=+24.274665320"
Sep 17 17:35:09.892961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:09.892925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:09.893136 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:09.893039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:10.033718 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:10.033664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp" event={"ID":"afd3ba62-6b4f-4a4f-adf9-4a3191c398ca","Type":"ContainerStarted","Data":"34fed3a7d68732bc1c412af2d7011703fd6e93664f7c716c01296db7ed16dd91"}
Sep 17 17:35:10.069016 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:10.068954    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2nkcp" podStartSLOduration=3.685022953 podStartE2EDuration="23.068934369s" totalImagesPullingTime="19.383911416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:10.067896244 +0000 UTC m=+25.299927065" watchObservedRunningTime="2026-09-17 17:35:10.068934369 +0000 UTC m=+25.300965189"
Sep 17 17:35:10.892516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:10.892478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:10.892694 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:10.892611    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:10.892694 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:10.892648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:10.892835 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:10.892742    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:11.039556 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:11.039530    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:35:11.040069 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:11.040029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"eb69b9f126bdf74ba430682cb1529ae655b1e04f0cbfcdde2910ed698b92f387"}
Sep 17 17:35:11.105492 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:11.105453    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:35:11.106129 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:11.106109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:35:11.892988 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:11.892825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:11.893151 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:11.893071    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:12.042442 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:12.042297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:35:12.042911 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:12.042895    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bfjrr"
Sep 17 17:35:12.892796 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:12.892720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:12.892942 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:12.892811    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:12.892942 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:12.892893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:12.893043 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:12.893012    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:13.049767 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:13.049743    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:35:13.050581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:13.050129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"7acab607546ca694ce2b1505e8ee480d39259cd053ac262de62ebc4d1bb789c7"}
Sep 17 17:35:13.050581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:13.050436    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:35:13.050581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:13.050493    2581 scope.go:117] "RemoveContainer" containerID="eaa45c6cf57bfa6d36e2ca0e23fe9dbc22fa047dd33f4518d51762c98fa35372"
Sep 17 17:35:13.052188 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:13.052166    2581 generic.go:356] "Generic (PLEG): container finished" podID="91affe17-634c-4ac8-b1a7-b60485e21db4" containerID="5f5e2dd5d691dde932260260f9b4e9f56f1830995fe1d9bc0e46e0b190e265ab" exitCode=0
Sep 17 17:35:13.052311 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:13.052207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krjlf" event={"ID":"91affe17-634c-4ac8-b1a7-b60485e21db4","Type":"ContainerDied","Data":"5f5e2dd5d691dde932260260f9b4e9f56f1830995fe1d9bc0e46e0b190e265ab"}
Sep 17 17:35:13.066775 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:13.066756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:35:13.892521 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:13.892496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:13.892692 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:13.892601    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:14.057957 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.057936    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:35:14.058346 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.058320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" event={"ID":"fb65f3ad-31a3-4b27-b383-8859526e81d4","Type":"ContainerStarted","Data":"cd4027cdef8ec9061a1805b4a284aa9d01a27007889e9c1bdc3644f212fc9682"}
Sep 17 17:35:14.058491 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.058474    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:35:14.058730 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.058699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:35:14.072882 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.072858    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:35:14.091815 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.091770    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" podStartSLOduration=9.524140883 podStartE2EDuration="27.091755056s" totalImagesPullingTime="17.567614173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:14.091410684 +0000 UTC m=+29.323441504" watchObservedRunningTime="2026-09-17 17:35:14.091755056 +0000 UTC m=+29.323785875"
Sep 17 17:35:14.466555 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.466528    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2bl7r"]
Sep 17 17:35:14.466719 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.466684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:14.466815 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:14.466794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:14.470192 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.470150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kc6h8"]
Sep 17 17:35:14.470295 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.470284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:14.470394 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:14.470377    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:14.470804 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.470782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5225q"]
Sep 17 17:35:14.470926 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:14.470881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:14.470997 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:14.470974    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:15.062741 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:15.062687    2581 generic.go:356] "Generic (PLEG): container finished" podID="91affe17-634c-4ac8-b1a7-b60485e21db4" containerID="bcbe4a85b01d187b33d2097cd498ab95a00620cd755bb21d79a5b3a6b1a4049e" exitCode=0
Sep 17 17:35:15.063237 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:15.062781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krjlf" event={"ID":"91affe17-634c-4ac8-b1a7-b60485e21db4","Type":"ContainerDied","Data":"bcbe4a85b01d187b33d2097cd498ab95a00620cd755bb21d79a5b3a6b1a4049e"}
Sep 17 17:35:15.063237 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:15.062868    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:35:15.892781 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:15.892752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:15.892961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:15.892758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:15.892961 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:15.892847    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:15.892961 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:15.892930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:16.065363 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:16.065338    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:35:16.894026 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:16.893841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:16.894172 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:16.894038    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:17.069473 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:17.069439    2581 generic.go:356] "Generic (PLEG): container finished" podID="91affe17-634c-4ac8-b1a7-b60485e21db4" containerID="2ec1171ec550e9b48af5271d93aed8076faeb2f4d2756b2f2248d45e8c8a1322" exitCode=0
Sep 17 17:35:17.069821 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:17.069494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krjlf" event={"ID":"91affe17-634c-4ac8-b1a7-b60485e21db4","Type":"ContainerDied","Data":"2ec1171ec550e9b48af5271d93aed8076faeb2f4d2756b2f2248d45e8c8a1322"}
Sep 17 17:35:17.892518 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:17.892483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:17.892728 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:17.892521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:17.892728 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:17.892625    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:17.892850 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:17.892785    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:18.323982 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:18.323938    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:35:18.324387 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:18.324311    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:35:18.340195 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:18.340144    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" podUID="fb65f3ad-31a3-4b27-b383-8859526e81d4" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 17:35:18.349574 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:18.349542    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr" podUID="fb65f3ad-31a3-4b27-b383-8859526e81d4" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 17:35:18.893062 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:18.893028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:18.893240 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:18.893150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:19.892664 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:19.892633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:19.893178 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:19.892769    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kc6h8" podUID="b665814a-b3dd-4590-9dbf-63559f1f5681"
Sep 17 17:35:19.893178 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:19.892802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:19.893290 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:19.893232    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bl7r" podUID="6d832701-c936-4fba-b54d-df1ed6594c63"
Sep 17 17:35:20.477896 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:20.477853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:20.478080 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.478029    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:20.478142 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.478120    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs podName:b665814a-b3dd-4590-9dbf-63559f1f5681 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:52.478099353 +0000 UTC m=+67.710130149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs") pod "network-metrics-daemon-kc6h8" (UID: "b665814a-b3dd-4590-9dbf-63559f1f5681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:20.578197 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:20.578164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:20.578413 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.578313    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:20.578413 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.578385    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret podName:6d832701-c936-4fba-b54d-df1ed6594c63 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:52.57836902 +0000 UTC m=+67.810399825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret") pod "global-pull-secret-syncer-2bl7r" (UID: "6d832701-c936-4fba-b54d-df1ed6594c63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:20.679336 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:20.679261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:20.679474 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.679381    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:20.679474 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.679401    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:20.679474 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.679411    2581 projected.go:196] Error preparing data for projected volume kube-api-access-j9w6b for pod openshift-network-diagnostics/network-check-target-5225q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:20.679474 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.679464    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b podName:db4b0ca9-0163-42ec-be91-e396a0cdaf9f nodeName:}" failed. No retries permitted until 2026-09-17 17:35:52.67944936 +0000 UTC m=+67.911480157 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9w6b" (UniqueName: "kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b") pod "network-check-target-5225q" (UID: "db4b0ca9-0163-42ec-be91-e396a0cdaf9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:20.893416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:20.893378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:20.893850 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:20.893508    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5225q" podUID="db4b0ca9-0163-42ec-be91-e396a0cdaf9f"
Sep 17 17:35:21.109892 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.109861    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeReady"
Sep 17 17:35:21.110054 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.110021    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:35:21.185871 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.185845    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zmpdc"]
Sep 17 17:35:21.199526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.199501    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9kcf6"]
Sep 17 17:35:21.199643 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.199596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.202172 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.202157    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gbrs5\""
Sep 17 17:35:21.202172 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.202161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:35:21.202328 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.202201    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:35:21.219714 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.219688    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmpdc"]
Sep 17 17:35:21.219808 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.219738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9kcf6"]
Sep 17 17:35:21.219808 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.219801    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9kcf6"
Sep 17 17:35:21.222203 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.222185    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:35:21.222297 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.222252    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:35:21.222297 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.222269    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:35:21.222381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.222308    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2tbmw\""
Sep 17 17:35:21.283912 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.283845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbf5bec-f51d-44ff-ba63-a47812c3b733-config-volume\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.283912 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.283895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71-cert\") pod \"ingress-canary-9kcf6\" (UID: \"27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71\") " pod="openshift-ingress-canary/ingress-canary-9kcf6"
Sep 17 17:35:21.284062 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.283947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbf5bec-f51d-44ff-ba63-a47812c3b733-tmp-dir\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.284062 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.283983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.284062 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.284033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57br\" (UniqueName: \"kubernetes.io/projected/ecbf5bec-f51d-44ff-ba63-a47812c3b733-kube-api-access-x57br\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.284155 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.284059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqpqw\" (UniqueName: \"kubernetes.io/projected/27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71-kube-api-access-pqpqw\") pod \"ingress-canary-9kcf6\" (UID: \"27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71\") " pod="openshift-ingress-canary/ingress-canary-9kcf6"
Sep 17 17:35:21.384337 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.384310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbf5bec-f51d-44ff-ba63-a47812c3b733-config-volume\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.384569 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.384357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71-cert\") pod \"ingress-canary-9kcf6\" (UID: \"27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71\") " pod="openshift-ingress-canary/ingress-canary-9kcf6"
Sep 17 17:35:21.384569 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.384385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbf5bec-f51d-44ff-ba63-a47812c3b733-tmp-dir\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.384569 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.384411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.384569 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.384459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57br\" (UniqueName: \"kubernetes.io/projected/ecbf5bec-f51d-44ff-ba63-a47812c3b733-kube-api-access-x57br\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.384569 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.384488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqpqw\" (UniqueName: \"kubernetes.io/projected/27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71-kube-api-access-pqpqw\") pod \"ingress-canary-9kcf6\" (UID: \"27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71\") " pod="openshift-ingress-canary/ingress-canary-9kcf6"
Sep 17 17:35:21.384569 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:21.384553    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:21.384902 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:21.384620    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls podName:ecbf5bec-f51d-44ff-ba63-a47812c3b733 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:21.884605588 +0000 UTC m=+37.116636384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls") pod "dns-default-zmpdc" (UID: "ecbf5bec-f51d-44ff-ba63-a47812c3b733") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:21.384902 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.384740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbf5bec-f51d-44ff-ba63-a47812c3b733-tmp-dir\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.385050 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.385028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbf5bec-f51d-44ff-ba63-a47812c3b733-config-volume\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.388298 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.388280    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71-cert\") pod \"ingress-canary-9kcf6\" (UID: \"27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71\") " pod="openshift-ingress-canary/ingress-canary-9kcf6"
Sep 17 17:35:21.399895 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.399874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqpqw\" (UniqueName: \"kubernetes.io/projected/27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71-kube-api-access-pqpqw\") pod \"ingress-canary-9kcf6\" (UID: \"27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71\") " pod="openshift-ingress-canary/ingress-canary-9kcf6"
Sep 17 17:35:21.399895 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.399883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57br\" (UniqueName: \"kubernetes.io/projected/ecbf5bec-f51d-44ff-ba63-a47812c3b733-kube-api-access-x57br\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.527752 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.527729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9kcf6"
Sep 17 17:35:21.803966 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.803889    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9kcf6"]
Sep 17 17:35:21.807509 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:35:21.807483    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b7bd4d_0c64_42ff_8ad6_a5a9d5a9ee71.slice/crio-33a9bfa3166568ca561f87dc39679e7c36c0b0e59d489f04358980ba5d5a7ba9 WatchSource:0}: Error finding container 33a9bfa3166568ca561f87dc39679e7c36c0b0e59d489f04358980ba5d5a7ba9: Status 404 returned error can't find the container with id 33a9bfa3166568ca561f87dc39679e7c36c0b0e59d489f04358980ba5d5a7ba9
Sep 17 17:35:21.888866 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.888839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:21.888954 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:21.888942    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:21.889005 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:21.888996    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls podName:ecbf5bec-f51d-44ff-ba63-a47812c3b733 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.888981253 +0000 UTC m=+38.121012050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls") pod "dns-default-zmpdc" (UID: "ecbf5bec-f51d-44ff-ba63-a47812c3b733") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:21.892850 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.892828    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:21.893000 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.892984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:21.895935 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.895915    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:21.896516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.895991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:21.896516 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:21.896036    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xc26q\""
Sep 17 17:35:22.086840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:22.086719    2581 generic.go:356] "Generic (PLEG): container finished" podID="91affe17-634c-4ac8-b1a7-b60485e21db4" containerID="ec64a85ccb39fe09af193bfc4755561418065ad779977a7fcb920006424a323f" exitCode=0
Sep 17 17:35:22.086840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:22.086738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krjlf" event={"ID":"91affe17-634c-4ac8-b1a7-b60485e21db4","Type":"ContainerDied","Data":"ec64a85ccb39fe09af193bfc4755561418065ad779977a7fcb920006424a323f"}
Sep 17 17:35:22.087941 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:22.087921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9kcf6" event={"ID":"27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71","Type":"ContainerStarted","Data":"33a9bfa3166568ca561f87dc39679e7c36c0b0e59d489f04358980ba5d5a7ba9"}
Sep 17 17:35:22.892493 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:22.892463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:22.896333 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:22.896310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:22.896740 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:22.896496    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:22.896740 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:22.896556    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls podName:ecbf5bec-f51d-44ff-ba63-a47812c3b733 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.896537079 +0000 UTC m=+40.128567881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls") pod "dns-default-zmpdc" (UID: "ecbf5bec-f51d-44ff-ba63-a47812c3b733") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:22.897665 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:22.897640    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:22.897810 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:22.897743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:22.898830 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:22.898809    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d2vf4\""
Sep 17 17:35:23.094320 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:23.094280    2581 generic.go:356] "Generic (PLEG): container finished" podID="91affe17-634c-4ac8-b1a7-b60485e21db4" containerID="991369d8a938a3f7b8a11596df3e2bf9dcb2f92df1863f1f7e44f98b7cace198" exitCode=0
Sep 17 17:35:23.094491 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:23.094330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krjlf" event={"ID":"91affe17-634c-4ac8-b1a7-b60485e21db4","Type":"ContainerDied","Data":"991369d8a938a3f7b8a11596df3e2bf9dcb2f92df1863f1f7e44f98b7cace198"}
Sep 17 17:35:24.100936 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:24.100895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krjlf" event={"ID":"91affe17-634c-4ac8-b1a7-b60485e21db4","Type":"ContainerStarted","Data":"97e035c6dd7370ce1b43c7e058be6b3ee5b68749fd9c5a06f83344cdb6094f8e"}
Sep 17 17:35:24.126520 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:24.126474    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-krjlf" podStartSLOduration=8.416548043 podStartE2EDuration="37.126457665s" totalImagesPullingTime="28.709909622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:24.124908291 +0000 UTC m=+39.356939110" watchObservedRunningTime="2026-09-17 17:35:24.126457665 +0000 UTC m=+39.358488484"
Sep 17 17:35:24.911094 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:24.910933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:24.911094 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:24.911061    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:24.911295 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:24.911150    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls podName:ecbf5bec-f51d-44ff-ba63-a47812c3b733 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:28.911137101 +0000 UTC m=+44.143167898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls") pod "dns-default-zmpdc" (UID: "ecbf5bec-f51d-44ff-ba63-a47812c3b733") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:25.104561 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:25.104523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9kcf6" event={"ID":"27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71","Type":"ContainerStarted","Data":"0e19645839c54741dacd4f1f5584f1de1e1b72850602041c4581a35c6e5546bc"}
Sep 17 17:35:25.123572 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:25.123525    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9kcf6" podStartSLOduration=1.8499182809999999 podStartE2EDuration="4.12350997s" totalImagesPullingTime="2.273591689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:25.122452923 +0000 UTC m=+40.354483743" watchObservedRunningTime="2026-09-17 17:35:25.12350997 +0000 UTC m=+40.355540792"
Sep 17 17:35:28.934301 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:28.934274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:28.934772 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:28.934441    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:28.934772 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:28.934496    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls podName:ecbf5bec-f51d-44ff-ba63-a47812c3b733 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:36.934482972 +0000 UTC m=+52.166513780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls") pod "dns-default-zmpdc" (UID: "ecbf5bec-f51d-44ff-ba63-a47812c3b733") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:36.988911 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:36.988869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:36.989382 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:36.988984    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:36.989382 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:36.989046    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls podName:ecbf5bec-f51d-44ff-ba63-a47812c3b733 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:52.989032337 +0000 UTC m=+68.221063147 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls") pod "dns-default-zmpdc" (UID: "ecbf5bec-f51d-44ff-ba63-a47812c3b733") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:48.349510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:48.349483    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ch8qr"
Sep 17 17:35:52.490260 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.490222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:35:52.493339 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.493319    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:52.500973 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:52.500956    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:35:52.501046 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:52.501013    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs podName:b665814a-b3dd-4590-9dbf-63559f1f5681 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:56.50099882 +0000 UTC m=+131.733029617 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs") pod "network-metrics-daemon-kc6h8" (UID: "b665814a-b3dd-4590-9dbf-63559f1f5681") : secret "metrics-daemon-secret" not found
Sep 17 17:35:52.590984 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.590936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:52.593728 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.593697    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:52.605314 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.605289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6d832701-c936-4fba-b54d-df1ed6594c63-original-pull-secret\") pod \"global-pull-secret-syncer-2bl7r\" (UID: \"6d832701-c936-4fba-b54d-df1ed6594c63\") " pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:52.691795 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.691775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:52.694680 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.694661    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:52.705844 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.705823    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:52.715347 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.715327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9w6b\" (UniqueName: \"kubernetes.io/projected/db4b0ca9-0163-42ec-be91-e396a0cdaf9f-kube-api-access-j9w6b\") pod \"network-check-target-5225q\" (UID: \"db4b0ca9-0163-42ec-be91-e396a0cdaf9f\") " pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:52.809061 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.809037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bl7r"
Sep 17 17:35:52.906245 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.906219    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d2vf4\""
Sep 17 17:35:52.913783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.913765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:52.994113 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:52.994076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:35:52.994226 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:52.994195    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:52.994264 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:35:52.994256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls podName:ecbf5bec-f51d-44ff-ba63-a47812c3b733 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:24.994238489 +0000 UTC m=+100.226269287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls") pod "dns-default-zmpdc" (UID: "ecbf5bec-f51d-44ff-ba63-a47812c3b733") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:53.085884 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:53.085822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2bl7r"]
Sep 17 17:35:53.089044 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:35:53.089020    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d832701_c936_4fba_b54d_df1ed6594c63.slice/crio-cfcf3c5fc523d05dc7f71dbaa260add64b9cc62d0a6e425b0b1a2abc0b5f4452 WatchSource:0}: Error finding container cfcf3c5fc523d05dc7f71dbaa260add64b9cc62d0a6e425b0b1a2abc0b5f4452: Status 404 returned error can't find the container with id cfcf3c5fc523d05dc7f71dbaa260add64b9cc62d0a6e425b0b1a2abc0b5f4452
Sep 17 17:35:53.170296 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:53.170263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2bl7r" event={"ID":"6d832701-c936-4fba-b54d-df1ed6594c63","Type":"ContainerStarted","Data":"cfcf3c5fc523d05dc7f71dbaa260add64b9cc62d0a6e425b0b1a2abc0b5f4452"}
Sep 17 17:35:53.186033 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:53.186011    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5225q"]
Sep 17 17:35:53.189059 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:35:53.189038    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4b0ca9_0163_42ec_be91_e396a0cdaf9f.slice/crio-999200ac5e49677153bcb397b5184ad2883a321135fb0008facf08ed1df8b198 WatchSource:0}: Error finding container 999200ac5e49677153bcb397b5184ad2883a321135fb0008facf08ed1df8b198: Status 404 returned error can't find the container with id 999200ac5e49677153bcb397b5184ad2883a321135fb0008facf08ed1df8b198
Sep 17 17:35:54.174851 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:54.174773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5225q" event={"ID":"db4b0ca9-0163-42ec-be91-e396a0cdaf9f","Type":"ContainerStarted","Data":"999200ac5e49677153bcb397b5184ad2883a321135fb0008facf08ed1df8b198"}
Sep 17 17:35:58.186628 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:58.186587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2bl7r" event={"ID":"6d832701-c936-4fba-b54d-df1ed6594c63","Type":"ContainerStarted","Data":"2cd1a73f73c8d249eb15957b0a54e3c54498a4fdacb6c9ac32da5e1999f9955e"}
Sep 17 17:35:58.187991 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:58.187967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5225q" event={"ID":"db4b0ca9-0163-42ec-be91-e396a0cdaf9f","Type":"ContainerStarted","Data":"2d809c7cacc5f749f8a092891a583299fa56682800bca6799226919dcb4ecf10"}
Sep 17 17:35:58.188109 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:58.188098    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:35:58.204651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:58.204608    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2bl7r" podStartSLOduration=66.878396613 podStartE2EDuration="1m11.204591656s" totalImagesPullingTime="4.326195043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:58.204225167 +0000 UTC m=+73.436255987" watchObservedRunningTime="2026-09-17 17:35:58.204591656 +0000 UTC m=+73.436622476"
Sep 17 17:35:58.223209 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:35:58.223167    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5225q" podStartSLOduration=67.000518077 podStartE2EDuration="1m11.223157329s" totalImagesPullingTime="4.222639252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:58.223106256 +0000 UTC m=+73.455137088" watchObservedRunningTime="2026-09-17 17:35:58.223157329 +0000 UTC m=+73.455188149"
Sep 17 17:36:12.362442 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.362406    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-dfd2h"]
Sep 17 17:36:12.364967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.364951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.367724 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.367692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:36:12.367822 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.367794    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:36:12.368416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.368401    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:36:12.368910 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.368896    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t2dz9\""
Sep 17 17:36:12.370077 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.370058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:36:12.376754 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.376734    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:36:12.381029 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.381012    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-dfd2h"]
Sep 17 17:36:12.477990 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.477951    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54f86dc978-xcqrf"]
Sep 17 17:36:12.479683 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.479666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.489990 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.489969    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"]
Sep 17 17:36:12.491905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.491887    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"]
Sep 17 17:36:12.492023 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.491992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:12.493678 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.493661    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q"]
Sep 17 17:36:12.493795 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.493765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.495052 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.495038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:36:12.495327 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.495310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:36:12.495398 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.495360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q"
Sep 17 17:36:12.495741 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.495725    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:12.495797 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.495732    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mzr6j\""
Sep 17 17:36:12.496812 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.496793    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:36:12.496905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.496813    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:36:12.496905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.496843    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:36:12.499400 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.499346    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:36:12.499529 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.499514    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:12.502282 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.502264    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:12.503059 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.502997    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54f86dc978-xcqrf"]
Sep 17 17:36:12.503358 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.503338    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gww56\""
Sep 17 17:36:12.504548 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.504522    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:12.504879 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.504858    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cf4r8\""
Sep 17 17:36:12.505111 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.505090    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wggg7\""
Sep 17 17:36:12.507752 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.507730    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:36:12.509898 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.509878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:36:12.509979 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.509888    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:36:12.519208 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.519180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a506c7-09c6-4921-91b2-fd8ea4a044c8-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.519297 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.519217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a506c7-09c6-4921-91b2-fd8ea4a044c8-tmp\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.519297 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.519239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51a506c7-09c6-4921-91b2-fd8ea4a044c8-serving-cert\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.519367 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.519316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a506c7-09c6-4921-91b2-fd8ea4a044c8-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.519367 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.519349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hzpz\" (UniqueName: \"kubernetes.io/projected/51a506c7-09c6-4921-91b2-fd8ea4a044c8-kube-api-access-5hzpz\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.519430 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.519376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/51a506c7-09c6-4921-91b2-fd8ea4a044c8-snapshots\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.525488 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.525454    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"]
Sep 17 17:36:12.526152 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.526130    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q"]
Sep 17 17:36:12.531968 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.531948    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"]
Sep 17 17:36:12.599031 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.598997    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"]
Sep 17 17:36:12.601272 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.601219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:12.608262 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.608237    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-operator-config\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap"
Sep 17 17:36:12.609489 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.609468    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"serving-cert\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret"
Sep 17 17:36:12.612832 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.612794    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:12.612832 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.612815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7h5nd\""
Sep 17 17:36:12.613844 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.613831    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:12.620152 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-bound-sa-token\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.620210 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.620210 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.620210 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-trusted-ca\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.620303 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrlcl\" (UniqueName: \"kubernetes.io/projected/e197166b-a6d8-4e4b-acee-184344294481-kube-api-access-nrlcl\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:12.620303 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a506c7-09c6-4921-91b2-fd8ea4a044c8-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.620303 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fscgd\" (UniqueName: \"kubernetes.io/projected/68ec1a62-564e-403e-a5a3-85179a62f64f-kube-api-access-fscgd\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.620393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49bl9\" (UniqueName: \"kubernetes.io/projected/975b9941-2cf8-4a56-901c-72e4d0a0d7f8-kube-api-access-49bl9\") pod \"volume-data-source-validator-5c4db8bdf9-xzc7q\" (UID: \"975b9941-2cf8-4a56-901c-72e4d0a0d7f8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q"
Sep 17 17:36:12.620393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hzpz\" (UniqueName: \"kubernetes.io/projected/51a506c7-09c6-4921-91b2-fd8ea4a044c8-kube-api-access-5hzpz\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.620453 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/51a506c7-09c6-4921-91b2-fd8ea4a044c8-snapshots\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.620483 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-image-registry-private-configuration\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.620483 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c68274e4-1f0f-4801-b58b-a3af19bd5c91-ca-trust-extracted\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.620624 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-certificates\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.620624 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a506c7-09c6-4921-91b2-fd8ea4a044c8-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.620624 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a506c7-09c6-4921-91b2-fd8ea4a044c8-tmp\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.620742 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:12.620742 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-installation-pull-secrets\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.620742 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51a506c7-09c6-4921-91b2-fd8ea4a044c8-serving-cert\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.620861 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qftmx\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-kube-api-access-qftmx\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.620861 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.620763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/68ec1a62-564e-403e-a5a3-85179a62f64f-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.621084 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.621065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51a506c7-09c6-4921-91b2-fd8ea4a044c8-tmp\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.621140 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.621079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a506c7-09c6-4921-91b2-fd8ea4a044c8-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.621140 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.621090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/51a506c7-09c6-4921-91b2-fd8ea4a044c8-snapshots\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.621140 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.621101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a506c7-09c6-4921-91b2-fd8ea4a044c8-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.622916 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.622901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51a506c7-09c6-4921-91b2-fd8ea4a044c8-serving-cert\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.645500 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.645468    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"]
Sep 17 17:36:12.647032 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.647012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hzpz\" (UniqueName: \"kubernetes.io/projected/51a506c7-09c6-4921-91b2-fd8ea4a044c8-kube-api-access-5hzpz\") pod \"insights-operator-6f9c644dd9-dfd2h\" (UID: \"51a506c7-09c6-4921-91b2-fd8ea4a044c8\") " pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.674110 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.674082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h"
Sep 17 17:36:12.692271 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.692247    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8559889cdb-wd9h9"]
Sep 17 17:36:12.697039 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.697018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.700251 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.700231    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 17:36:12.701115 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.701098    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 17:36:12.701227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.701146    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 17:36:12.701227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.701186    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:36:12.701426 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.701410    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:36:12.701525 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.701510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sht57\""
Sep 17 17:36:12.701616 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.701603    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 17:36:12.712128 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.712068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8559889cdb-wd9h9"]
Sep 17 17:36:12.721662 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-trusted-ca\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.721662 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrlcl\" (UniqueName: \"kubernetes.io/projected/e197166b-a6d8-4e4b-acee-184344294481-kube-api-access-nrlcl\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:12.721840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b7c5595-dde6-4720-bb6d-7633122dc737-config\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:12.721840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-image-registry-private-configuration\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.721840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-certificates\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.721840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:12.721840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qftmx\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-kube-api-access-qftmx\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.722087 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/68ec1a62-564e-403e-a5a3-85179a62f64f-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.722087 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.722087 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49bl9\" (UniqueName: \"kubernetes.io/projected/975b9941-2cf8-4a56-901c-72e4d0a0d7f8-kube-api-access-49bl9\") pod \"volume-data-source-validator-5c4db8bdf9-xzc7q\" (UID: \"975b9941-2cf8-4a56-901c-72e4d0a0d7f8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q"
Sep 17 17:36:12.722087 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c68274e4-1f0f-4801-b58b-a3af19bd5c91-ca-trust-extracted\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.722087 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.721976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2974\" (UniqueName: \"kubernetes.io/projected/1b7c5595-dde6-4720-bb6d-7633122dc737-kube-api-access-k2974\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:12.722087 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.722077    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:12.722355 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.722189    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls podName:e197166b-a6d8-4e4b-acee-184344294481 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:13.222130612 +0000 UTC m=+88.454161430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-xzmx9" (UID: "e197166b-a6d8-4e4b-acee-184344294481") : secret "samples-operator-tls" not found
Sep 17 17:36:12.722460 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.722419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-certificates\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.722460 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.722081    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:12.722597 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.722463    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54f86dc978-xcqrf: secret "image-registry-tls" not found
Sep 17 17:36:12.722597 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.722549    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls podName:c68274e4-1f0f-4801-b58b-a3af19bd5c91 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:13.222532372 +0000 UTC m=+88.454563189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls") pod "image-registry-54f86dc978-xcqrf" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91") : secret "image-registry-tls" not found
Sep 17 17:36:12.722597 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.722550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c68274e4-1f0f-4801-b58b-a3af19bd5c91-ca-trust-extracted\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.722597 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.722571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-installation-pull-secrets\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.722750 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.722611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fscgd\" (UniqueName: \"kubernetes.io/projected/68ec1a62-564e-403e-a5a3-85179a62f64f-kube-api-access-fscgd\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.722750 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.722651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-bound-sa-token\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.722750 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.722680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b7c5595-dde6-4720-bb6d-7633122dc737-serving-cert\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:12.722750 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.722738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.722869 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.722846    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:12.722903 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.722892    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls podName:68ec1a62-564e-403e-a5a3-85179a62f64f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:13.222877261 +0000 UTC m=+88.454908071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xlltt" (UID: "68ec1a62-564e-403e-a5a3-85179a62f64f") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:12.722966 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.722911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-trusted-ca\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.723297 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.723264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/68ec1a62-564e-403e-a5a3-85179a62f64f-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.725671 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.725633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-image-registry-private-configuration\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.727004 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.726899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-installation-pull-secrets\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.738856 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.738831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49bl9\" (UniqueName: \"kubernetes.io/projected/975b9941-2cf8-4a56-901c-72e4d0a0d7f8-kube-api-access-49bl9\") pod \"volume-data-source-validator-5c4db8bdf9-xzc7q\" (UID: \"975b9941-2cf8-4a56-901c-72e4d0a0d7f8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q"
Sep 17 17:36:12.739587 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.739571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qftmx\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-kube-api-access-qftmx\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.743125 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.743105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fscgd\" (UniqueName: \"kubernetes.io/projected/68ec1a62-564e-403e-a5a3-85179a62f64f-kube-api-access-fscgd\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:12.746063 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.746042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrlcl\" (UniqueName: \"kubernetes.io/projected/e197166b-a6d8-4e4b-acee-184344294481-kube-api-access-nrlcl\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:12.747223 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.747188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-bound-sa-token\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:12.814597 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.814577    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q"
Sep 17 17:36:12.823382 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.823360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b7c5595-dde6-4720-bb6d-7633122dc737-serving-cert\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:12.823443 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.823421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b7c5595-dde6-4720-bb6d-7633122dc737-config\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:12.823506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.823487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-default-certificate\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.823556 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.823525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s269h\" (UniqueName: \"kubernetes.io/projected/71a9e678-1ce7-4e6e-b8bc-6140a7984222-kube-api-access-s269h\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.823619 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.823599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.823668 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.823636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-stats-auth\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.823668 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.823653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.823755 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.823674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2974\" (UniqueName: \"kubernetes.io/projected/1b7c5595-dde6-4720-bb6d-7633122dc737-kube-api-access-k2974\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:12.835130 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.835109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2974\" (UniqueName: \"kubernetes.io/projected/1b7c5595-dde6-4720-bb6d-7633122dc737-kube-api-access-k2974\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:12.924698 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.924627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-default-certificate\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.924698 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.924664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s269h\" (UniqueName: \"kubernetes.io/projected/71a9e678-1ce7-4e6e-b8bc-6140a7984222-kube-api-access-s269h\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.924936 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.924727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.924936 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.924750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-stats-auth\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.924936 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.924765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.924936 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.924870    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:12.924936 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.924899    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:13.424876177 +0000 UTC m=+88.656906975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:12.924936 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:12.924928    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:13.424916971 +0000 UTC m=+88.656947773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : secret "router-metrics-certs-default" not found
Sep 17 17:36:12.927056 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.927034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-default-certificate\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.927165 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.927059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-stats-auth\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.942439 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.942412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s269h\" (UniqueName: \"kubernetes.io/projected/71a9e678-1ce7-4e6e-b8bc-6140a7984222-kube-api-access-s269h\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:12.955441 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:12.955415    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-dfd2h"]
Sep 17 17:36:12.958762 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:12.958736    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a506c7_09c6_4921_91b2_fd8ea4a044c8.slice/crio-4fcc68ea3bd46b0841e00c756351b05b00a8690e681de32052bcdb1055d83f01 WatchSource:0}: Error finding container 4fcc68ea3bd46b0841e00c756351b05b00a8690e681de32052bcdb1055d83f01: Status 404 returned error can't find the container with id 4fcc68ea3bd46b0841e00c756351b05b00a8690e681de32052bcdb1055d83f01
Sep 17 17:36:13.089866 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.089839    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q"]
Sep 17 17:36:13.092577 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:13.092551    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975b9941_2cf8_4a56_901c_72e4d0a0d7f8.slice/crio-cbab83696be55203ca33a5a8887bef1eb7cf27cff81fd50e198bca5ca1a55f5c WatchSource:0}: Error finding container cbab83696be55203ca33a5a8887bef1eb7cf27cff81fd50e198bca5ca1a55f5c: Status 404 returned error can't find the container with id cbab83696be55203ca33a5a8887bef1eb7cf27cff81fd50e198bca5ca1a55f5c
Sep 17 17:36:13.224021 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.223949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q" event={"ID":"975b9941-2cf8-4a56-901c-72e4d0a0d7f8","Type":"ContainerStarted","Data":"cbab83696be55203ca33a5a8887bef1eb7cf27cff81fd50e198bca5ca1a55f5c"}
Sep 17 17:36:13.225061 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.225040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h" event={"ID":"51a506c7-09c6-4921-91b2-fd8ea4a044c8","Type":"ContainerStarted","Data":"4fcc68ea3bd46b0841e00c756351b05b00a8690e681de32052bcdb1055d83f01"}
Sep 17 17:36:13.227398 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.227380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:13.227460 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.227444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:13.227499 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.227477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:13.227556 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.227538    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:13.227556 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.227552    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:13.227627 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.227560    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54f86dc978-xcqrf: secret "image-registry-tls" not found
Sep 17 17:36:13.227627 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.227598    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls podName:e197166b-a6d8-4e4b-acee-184344294481 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:14.22758514 +0000 UTC m=+89.459615937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-xzmx9" (UID: "e197166b-a6d8-4e4b-acee-184344294481") : secret "samples-operator-tls" not found
Sep 17 17:36:13.227627 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.227606    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:13.227627 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.227620    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls podName:c68274e4-1f0f-4801-b58b-a3af19bd5c91 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:14.227603018 +0000 UTC m=+89.459633839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls") pod "image-registry-54f86dc978-xcqrf" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91") : secret "image-registry-tls" not found
Sep 17 17:36:13.227785 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.227658    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls podName:68ec1a62-564e-403e-a5a3-85179a62f64f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:14.227645959 +0000 UTC m=+89.459676756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xlltt" (UID: "68ec1a62-564e-403e-a5a3-85179a62f64f") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:13.428978 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.428945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:13.429337 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.429046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:13.429337 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.429077    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:13.429337 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.429143    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:14.429128889 +0000 UTC m=+89.661159686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : secret "router-metrics-certs-default" not found
Sep 17 17:36:13.429337 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.429157    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:14.429150818 +0000 UTC m=+89.661181615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:13.683093 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.683061    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:36:13.684510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.684483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b7c5595-dde6-4720-bb6d-7633122dc737-config\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:13.823890 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.823855    2581 secret.go:189] Couldn't get secret openshift-service-ca-operator/serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 17 17:36:13.824067 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:13.823954    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b7c5595-dde6-4720-bb6d-7633122dc737-serving-cert podName:1b7c5595-dde6-4720-bb6d-7633122dc737 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:14.323927163 +0000 UTC m=+89.555957965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/1b7c5595-dde6-4720-bb6d-7633122dc737-serving-cert") pod "service-ca-operator-57cdc4b768-jp46q" (UID: "1b7c5595-dde6-4720-bb6d-7633122dc737") : failed to sync secret cache: timed out waiting for the condition
Sep 17 17:36:13.838093 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:13.838067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:36:14.234275 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.234257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:14.234411 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.234397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:14.234454 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.234404    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:14.234454 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.234422    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54f86dc978-xcqrf: secret "image-registry-tls" not found
Sep 17 17:36:14.234454 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.234436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:14.234542 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.234475    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls podName:c68274e4-1f0f-4801-b58b-a3af19bd5c91 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:16.234457061 +0000 UTC m=+91.466487860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls") pod "image-registry-54f86dc978-xcqrf" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91") : secret "image-registry-tls" not found
Sep 17 17:36:14.234542 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.234520    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:14.234607 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.234558    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls podName:e197166b-a6d8-4e4b-acee-184344294481 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:16.234547807 +0000 UTC m=+91.466578604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-xzmx9" (UID: "e197166b-a6d8-4e4b-acee-184344294481") : secret "samples-operator-tls" not found
Sep 17 17:36:14.234607 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.234520    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:14.234607 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.234584    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls podName:68ec1a62-564e-403e-a5a3-85179a62f64f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:16.234578308 +0000 UTC m=+91.466609105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xlltt" (UID: "68ec1a62-564e-403e-a5a3-85179a62f64f") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:14.335622 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.335590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b7c5595-dde6-4720-bb6d-7633122dc737-serving-cert\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:14.337885 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.337862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b7c5595-dde6-4720-bb6d-7633122dc737-serving-cert\") pod \"service-ca-operator-57cdc4b768-jp46q\" (UID: \"1b7c5595-dde6-4720-bb6d-7633122dc737\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:14.410176 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.410154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"
Sep 17 17:36:14.436297 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.436259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:14.436631 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.436310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:14.436631 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.436441    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:16.436421733 +0000 UTC m=+91.668452543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:14.436631 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.436450    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:14.436631 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:14.436482    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:16.436472771 +0000 UTC m=+91.668503573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : secret "router-metrics-certs-default" not found
Sep 17 17:36:14.689625 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:14.689596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q"]
Sep 17 17:36:14.929993 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:14.929953    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7c5595_dde6_4720_bb6d_7633122dc737.slice/crio-4600de8aa36e41951514cd90ec1f18ce11cf41c6879f01b7ac264b4981c335c7 WatchSource:0}: Error finding container 4600de8aa36e41951514cd90ec1f18ce11cf41c6879f01b7ac264b4981c335c7: Status 404 returned error can't find the container with id 4600de8aa36e41951514cd90ec1f18ce11cf41c6879f01b7ac264b4981c335c7
Sep 17 17:36:15.231565 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.231479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q" event={"ID":"975b9941-2cf8-4a56-901c-72e4d0a0d7f8","Type":"ContainerStarted","Data":"8ded8f986aa3e27e206b6414e8a8a3829ea3a1e67a5fe38b42584885332edeac"}
Sep 17 17:36:15.233038 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.233013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h" event={"ID":"51a506c7-09c6-4921-91b2-fd8ea4a044c8","Type":"ContainerStarted","Data":"02fdb661d0095a9db3cee7b758bb5e56786ef18d4ef454f0186814169fa10bf5"}
Sep 17 17:36:15.234061 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.234043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q" event={"ID":"1b7c5595-dde6-4720-bb6d-7633122dc737","Type":"ContainerStarted","Data":"4600de8aa36e41951514cd90ec1f18ce11cf41c6879f01b7ac264b4981c335c7"}
Sep 17 17:36:15.252745 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.252685    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-xzc7q" podStartSLOduration=2.114780933 podStartE2EDuration="3.252670265s" totalImagesPullingTime="1.137889332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:15.252599083 +0000 UTC m=+90.484629901" watchObservedRunningTime="2026-09-17 17:36:15.252670265 +0000 UTC m=+90.484701082"
Sep 17 17:36:15.275744 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.275682    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h" podStartSLOduration=1.279620253 podStartE2EDuration="3.275670636s" totalImagesPullingTime="1.996050383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:15.274242505 +0000 UTC m=+90.506273323" watchObservedRunningTime="2026-09-17 17:36:15.275670636 +0000 UTC m=+90.507701455"
Sep 17 17:36:15.596217 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.596174    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dz2vf"]
Sep 17 17:36:15.599461 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.599431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.602647 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.602427    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:36:15.602647 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.602475    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:36:15.602837 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.602650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xtzf6\""
Sep 17 17:36:15.613606 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.613584    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dz2vf"]
Sep 17 17:36:15.748236 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.748205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44e82ca8-6bf7-44a5-96aa-3c28ba930259-data-volume\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.748438 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.748415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.748515 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.748491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44e82ca8-6bf7-44a5-96aa-3c28ba930259-crio-socket\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.748568 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.748553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.748621 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.748579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45xgj\" (UniqueName: \"kubernetes.io/projected/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-api-access-45xgj\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.848958 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.848878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.849118 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.848958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44e82ca8-6bf7-44a5-96aa-3c28ba930259-crio-socket\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.849118 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.849001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.849118 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:15.849026    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:15.849118 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:15.849086    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:15.849118 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.849087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44e82ca8-6bf7-44a5-96aa-3c28ba930259-crio-socket\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.849118 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:15.849098    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:16.349082076 +0000 UTC m=+91.581112873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:15.849389 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:15.849136    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:16.349118979 +0000 UTC m=+91.581149778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:15.849389 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.849155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45xgj\" (UniqueName: \"kubernetes.io/projected/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-api-access-45xgj\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.849555 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.849531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44e82ca8-6bf7-44a5-96aa-3c28ba930259-data-volume\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.849909 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.849888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44e82ca8-6bf7-44a5-96aa-3c28ba930259-data-volume\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:15.859159 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:15.859136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45xgj\" (UniqueName: \"kubernetes.io/projected/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-api-access-45xgj\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:16.253047 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:16.253002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:16.253047 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:16.253068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:16.253520 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.253132    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:16.253520 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:16.253178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:16.253520 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.253182    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:16.253520 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.253200    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls podName:e197166b-a6d8-4e4b-acee-184344294481 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.253181704 +0000 UTC m=+95.485212515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-xzmx9" (UID: "e197166b-a6d8-4e4b-acee-184344294481") : secret "samples-operator-tls" not found
Sep 17 17:36:16.253520 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.253230    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:16.253520 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.253201    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54f86dc978-xcqrf: secret "image-registry-tls" not found
Sep 17 17:36:16.253520 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.253273    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls podName:68ec1a62-564e-403e-a5a3-85179a62f64f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.253261058 +0000 UTC m=+95.485291859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xlltt" (UID: "68ec1a62-564e-403e-a5a3-85179a62f64f") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:16.253520 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.253299    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls podName:c68274e4-1f0f-4801-b58b-a3af19bd5c91 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.253288336 +0000 UTC m=+95.485319135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls") pod "image-registry-54f86dc978-xcqrf" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91") : secret "image-registry-tls" not found
Sep 17 17:36:16.353858 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:16.353823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:16.354280 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:16.354235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:16.354412 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.354303    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:16.354412 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.354345    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:16.354412 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.354396    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:17.354381634 +0000 UTC m=+92.586412448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:16.354542 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.354416    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:17.354406908 +0000 UTC m=+92.586437710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:16.455859 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:16.455817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:16.456028 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:16.455880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:16.456028 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.455955    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.455932869 +0000 UTC m=+95.687963681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:16.456028 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.456001    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:16.456162 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:16.456050    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.456036613 +0000 UTC m=+95.688067411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : secret "router-metrics-certs-default" not found
Sep 17 17:36:17.242232 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:17.242147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q" event={"ID":"1b7c5595-dde6-4720-bb6d-7633122dc737","Type":"ContainerStarted","Data":"4647fd8b58b21fe7c0a8ba00c7b503b66a229ced4f7bd7b6c96a00c70c3502ed"}
Sep 17 17:36:17.282109 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:17.282063    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q" podStartSLOduration=3.402282018 podStartE2EDuration="5.282047763s" totalImagesPullingTime="1.879765745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:17.280341035 +0000 UTC m=+92.512371855" watchObservedRunningTime="2026-09-17 17:36:17.282047763 +0000 UTC m=+92.514078582"
Sep 17 17:36:17.362019 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:17.361989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:17.362168 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:17.362083    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:17.362168 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:17.362123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:17.362168 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:17.362151    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:19.362136499 +0000 UTC m=+94.594167315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:17.362347 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:17.362171    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:17.362347 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:17.362214    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:19.362200694 +0000 UTC m=+94.594231508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:17.734210 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:17.734182    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbsv8_753a888b-2139-4ac2-afbf-2085bf6df478/dns-node-resolver/0.log"
Sep 17 17:36:18.551759 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:18.551734    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jjc62_fa4430ae-bc53-43aa-bc0e-77cb787e0d32/node-ca/0.log"
Sep 17 17:36:19.138431 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:19.138405    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9kcf6_27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71/serve-healthcheck-canary/0.log"
Sep 17 17:36:19.378027 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:19.377990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:19.378207 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:19.378061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:19.378207 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:19.378139    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:19.378331 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:19.378216    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:23.378195231 +0000 UTC m=+98.610226041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:19.378331 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:19.378138    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:19.378331 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:19.378286    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:23.378274054 +0000 UTC m=+98.610304851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:20.284874 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:20.284846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:20.284886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:20.284974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.285026    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.285052    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.285057    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.285074    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54f86dc978-xcqrf: secret "image-registry-tls" not found
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.285101    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls podName:e197166b-a6d8-4e4b-acee-184344294481 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.285084781 +0000 UTC m=+103.517115585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-xzmx9" (UID: "e197166b-a6d8-4e4b-acee-184344294481") : secret "samples-operator-tls" not found
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.285115    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls podName:c68274e4-1f0f-4801-b58b-a3af19bd5c91 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.285108158 +0000 UTC m=+103.517138954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls") pod "image-registry-54f86dc978-xcqrf" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91") : secret "image-registry-tls" not found
Sep 17 17:36:20.285234 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.285124    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls podName:68ec1a62-564e-403e-a5a3-85179a62f64f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.285119239 +0000 UTC m=+103.517150036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xlltt" (UID: "68ec1a62-564e-403e-a5a3-85179a62f64f") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:20.486314 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:20.486281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:20.486314 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:20.486323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:20.486532 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.486458    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.486438961 +0000 UTC m=+103.718469758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:20.486532 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.486461    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:20.486532 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:20.486496    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.486489904 +0000 UTC m=+103.718520701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : secret "router-metrics-certs-default" not found
Sep 17 17:36:23.407754 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:23.407716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:23.408183 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:23.407835    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.408183 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:23.407892    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:31.407877572 +0000 UTC m=+106.639908373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.408183 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:23.407839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:23.408183 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:23.407936    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:23.408183 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:23.408003    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:31.407988659 +0000 UTC m=+106.640019469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:25.019078 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:25.019039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:36:25.019557 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:25.019188    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:25.019557 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:25.019254    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls podName:ecbf5bec-f51d-44ff-ba63-a47812c3b733 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:29.019237109 +0000 UTC m=+164.251267909 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls") pod "dns-default-zmpdc" (UID: "ecbf5bec-f51d-44ff-ba63-a47812c3b733") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:28.343066 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.343022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:28.343066 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.343077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:28.343574 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.343153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:28.343574 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:28.343263    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:28.343574 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:28.343345    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls podName:68ec1a62-564e-403e-a5a3-85179a62f64f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:44.343326118 +0000 UTC m=+119.575356914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xlltt" (UID: "68ec1a62-564e-403e-a5a3-85179a62f64f") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:28.345816 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.345775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e197166b-a6d8-4e4b-acee-184344294481-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-xzmx9\" (UID: \"e197166b-a6d8-4e4b-acee-184344294481\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:28.345959 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.345815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"image-registry-54f86dc978-xcqrf\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") " pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:28.388168 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.388140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:28.401081 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.401058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"
Sep 17 17:36:28.545186 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.545146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:28.545325 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.545204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:28.545325 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:28.545315    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle podName:71a9e678-1ce7-4e6e-b8bc-6140a7984222 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:44.545296783 +0000 UTC m=+119.777327581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle") pod "router-default-8559889cdb-wd9h9" (UID: "71a9e678-1ce7-4e6e-b8bc-6140a7984222") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:28.547314 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.547291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a9e678-1ce7-4e6e-b8bc-6140a7984222-metrics-certs\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:28.666568 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.666508    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54f86dc978-xcqrf"]
Sep 17 17:36:28.669119 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:28.669092    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68274e4_1f0f_4801_b58b_a3af19bd5c91.slice/crio-86d263f9c789700f6c8177d23357ef2fe01a3111eab55cbd2c61e5db0029e837 WatchSource:0}: Error finding container 86d263f9c789700f6c8177d23357ef2fe01a3111eab55cbd2c61e5db0029e837: Status 404 returned error can't find the container with id 86d263f9c789700f6c8177d23357ef2fe01a3111eab55cbd2c61e5db0029e837
Sep 17 17:36:28.680838 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:28.680816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9"]
Sep 17 17:36:29.192047 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:29.192018    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5225q"
Sep 17 17:36:29.282851 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:29.282820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" event={"ID":"c68274e4-1f0f-4801-b58b-a3af19bd5c91","Type":"ContainerStarted","Data":"5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5"}
Sep 17 17:36:29.283016 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:29.282862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" event={"ID":"c68274e4-1f0f-4801-b58b-a3af19bd5c91","Type":"ContainerStarted","Data":"86d263f9c789700f6c8177d23357ef2fe01a3111eab55cbd2c61e5db0029e837"}
Sep 17 17:36:29.283016 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:29.282879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:36:29.283923 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:29.283904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9" event={"ID":"e197166b-a6d8-4e4b-acee-184344294481","Type":"ContainerStarted","Data":"bcc965639f0ddae90f159c6352be813e1e8c536f52019f82357232886ef980e5"}
Sep 17 17:36:29.306681 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:29.306643    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" podStartSLOduration=17.306629993 podStartE2EDuration="17.306629993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:29.305771396 +0000 UTC m=+104.537802208" watchObservedRunningTime="2026-09-17 17:36:29.306629993 +0000 UTC m=+104.538660811"
Sep 17 17:36:31.291193 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:31.291161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9" event={"ID":"e197166b-a6d8-4e4b-acee-184344294481","Type":"ContainerStarted","Data":"b37e976f35c5d3f92b68582d759a30c44609b590f9cd09399d31c6d733aca37c"}
Sep 17 17:36:31.291564 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:31.291209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9" event={"ID":"e197166b-a6d8-4e4b-acee-184344294481","Type":"ContainerStarted","Data":"9ae45492af10ca59a121dcd731af3b938cae000e053fd732c21db3bbd14d5a71"}
Sep 17 17:36:31.310133 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:31.310089    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-xzmx9" podStartSLOduration=17.106285707 podStartE2EDuration="19.310075329s" totalImagesPullingTime="2.203789622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:31.309774672 +0000 UTC m=+106.541805493" watchObservedRunningTime="2026-09-17 17:36:31.310075329 +0000 UTC m=+106.542106179"
Sep 17 17:36:31.470035 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:31.470010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:31.470146 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:31.470127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:31.470254 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:31.470240    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:31.470319 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:36:31.470304    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls podName:44e82ca8-6bf7-44a5-96aa-3c28ba930259 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:47.470286789 +0000 UTC m=+122.702317599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dz2vf" (UID: "44e82ca8-6bf7-44a5-96aa-3c28ba930259") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:31.470498 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:31.470476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44e82ca8-6bf7-44a5-96aa-3c28ba930259-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:44.361359 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.361317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:44.363862 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.363835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68ec1a62-564e-403e-a5a3-85179a62f64f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xlltt\" (UID: \"68ec1a62-564e-403e-a5a3-85179a62f64f\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:44.378836 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.378808    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"]
Sep 17 17:36:44.382258 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.382237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"
Sep 17 17:36:44.386660 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.386637    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:36:44.386793 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.386666    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mn2cz\""
Sep 17 17:36:44.386793 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.386646    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:36:44.397974 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.397942    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"]
Sep 17 17:36:44.425736 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.425698    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54f86dc978-xcqrf"]
Sep 17 17:36:44.430784 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.430763    2581 patch_prober.go:28] interesting pod/image-registry-54f86dc978-xcqrf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:36:44.430890 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.430818    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" podUID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:36:44.461889 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.461867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-b9f8t\" (UID: \"26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"
Sep 17 17:36:44.461979 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.461943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-b9f8t\" (UID: \"26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"
Sep 17 17:36:44.507292 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.507269    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b6df65cc4-jd6g9"]
Sep 17 17:36:44.511991 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.511969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.525120 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.525102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b6df65cc4-jd6g9"]
Sep 17 17:36:44.562227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-registry-tls\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.562227 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-trusted-ca\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.562381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-b9f8t\" (UID: \"26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"
Sep 17 17:36:44.562381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-ca-trust-extracted\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.562381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-registry-certificates\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.562381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-b9f8t\" (UID: \"26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"
Sep 17 17:36:44.562581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-bound-sa-token\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.562581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv8cg\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-kube-api-access-sv8cg\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.562581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-image-registry-private-configuration\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.562581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-installation-pull-secrets\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.562581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:44.562951 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.562934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-b9f8t\" (UID: \"26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"
Sep 17 17:36:44.563020 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.563004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a9e678-1ce7-4e6e-b8bc-6140a7984222-service-ca-bundle\") pod \"router-default-8559889cdb-wd9h9\" (UID: \"71a9e678-1ce7-4e6e-b8bc-6140a7984222\") " pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:44.564559 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.564538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-b9f8t\" (UID: \"26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"
Sep 17 17:36:44.610510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.610488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"
Sep 17 17:36:44.663159 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.663134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-bound-sa-token\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.663263 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.663170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv8cg\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-kube-api-access-sv8cg\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.663263 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.663204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-image-registry-private-configuration\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.663263 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.663235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-installation-pull-secrets\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.663432 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.663277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-registry-tls\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.663432 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.663309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-trusted-ca\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.663432 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.663346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-ca-trust-extracted\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.663432 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.663408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-registry-certificates\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.664070 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.664023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-ca-trust-extracted\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.664885 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.664490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-registry-certificates\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.664964 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.664952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-trusted-ca\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.666044 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.666016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-image-registry-private-configuration\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.667224 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.667195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-installation-pull-secrets\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.668651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.668612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-registry-tls\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.674255 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.674214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-bound-sa-token\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.674845 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.674805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv8cg\" (UniqueName: \"kubernetes.io/projected/e7307cdc-4814-4e1a-803c-1aad13e3ddb1-kube-api-access-sv8cg\") pod \"image-registry-6b6df65cc4-jd6g9\" (UID: \"e7307cdc-4814-4e1a-803c-1aad13e3ddb1\") " pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.691927 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.691872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"
Sep 17 17:36:44.806656 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.806631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:44.819744 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.819724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:44.884786 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.884597    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt"]
Sep 17 17:36:44.888231 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:44.888208    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ec1a62_564e_403e_a5a3_85179a62f64f.slice/crio-fa7c03a9fd515019425ee0805ceb7220e95ac47e3428f97537db78ab4b82e3dc WatchSource:0}: Error finding container fa7c03a9fd515019425ee0805ceb7220e95ac47e3428f97537db78ab4b82e3dc: Status 404 returned error can't find the container with id fa7c03a9fd515019425ee0805ceb7220e95ac47e3428f97537db78ab4b82e3dc
Sep 17 17:36:44.969081 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:44.969058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t"]
Sep 17 17:36:44.971077 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:44.971048    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a0a5e6_ffa0_43c2_be9c_0e1cd510bac3.slice/crio-21029e0d33903d7142d48dd38576601a20a04d49348925081c24ec725b821731 WatchSource:0}: Error finding container 21029e0d33903d7142d48dd38576601a20a04d49348925081c24ec725b821731: Status 404 returned error can't find the container with id 21029e0d33903d7142d48dd38576601a20a04d49348925081c24ec725b821731
Sep 17 17:36:45.078505 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.078484    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8559889cdb-wd9h9"]
Sep 17 17:36:45.081534 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:45.081513    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a9e678_1ce7_4e6e_b8bc_6140a7984222.slice/crio-35c4148f6a65ae6aa38f6b1d236ea1a31d61ad8d048e5c7f8a2196f883a94271 WatchSource:0}: Error finding container 35c4148f6a65ae6aa38f6b1d236ea1a31d61ad8d048e5c7f8a2196f883a94271: Status 404 returned error can't find the container with id 35c4148f6a65ae6aa38f6b1d236ea1a31d61ad8d048e5c7f8a2196f883a94271
Sep 17 17:36:45.102976 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.102955    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b6df65cc4-jd6g9"]
Sep 17 17:36:45.114528 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:45.114509    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7307cdc_4814_4e1a_803c_1aad13e3ddb1.slice/crio-f8185cfbbe4050ab818ae944051ad8b482cfdd0cd17529beacf32bb5c85478ef WatchSource:0}: Error finding container f8185cfbbe4050ab818ae944051ad8b482cfdd0cd17529beacf32bb5c85478ef: Status 404 returned error can't find the container with id f8185cfbbe4050ab818ae944051ad8b482cfdd0cd17529beacf32bb5c85478ef
Sep 17 17:36:45.342676 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.342637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8559889cdb-wd9h9" event={"ID":"71a9e678-1ce7-4e6e-b8bc-6140a7984222","Type":"ContainerStarted","Data":"914d4b99fb76cef383d264145f9a4d8881e634c614ecc52918d674be9533ded7"}
Sep 17 17:36:45.342862 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.342698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8559889cdb-wd9h9" event={"ID":"71a9e678-1ce7-4e6e-b8bc-6140a7984222","Type":"ContainerStarted","Data":"35c4148f6a65ae6aa38f6b1d236ea1a31d61ad8d048e5c7f8a2196f883a94271"}
Sep 17 17:36:45.344014 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.343985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt" event={"ID":"68ec1a62-564e-403e-a5a3-85179a62f64f","Type":"ContainerStarted","Data":"fa7c03a9fd515019425ee0805ceb7220e95ac47e3428f97537db78ab4b82e3dc"}
Sep 17 17:36:45.345668 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.345632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" event={"ID":"e7307cdc-4814-4e1a-803c-1aad13e3ddb1","Type":"ContainerStarted","Data":"31e18894ec4a7945077a5a7bfd33827a364f8e1ca78fd083bd04c9a94fcb33a5"}
Sep 17 17:36:45.345805 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.345695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:36:45.345805 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.345738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" event={"ID":"e7307cdc-4814-4e1a-803c-1aad13e3ddb1","Type":"ContainerStarted","Data":"f8185cfbbe4050ab818ae944051ad8b482cfdd0cd17529beacf32bb5c85478ef"}
Sep 17 17:36:45.346975 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.346935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t" event={"ID":"26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3","Type":"ContainerStarted","Data":"21029e0d33903d7142d48dd38576601a20a04d49348925081c24ec725b821731"}
Sep 17 17:36:45.367601 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.367558    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8559889cdb-wd9h9" podStartSLOduration=33.367543722 podStartE2EDuration="33.367543722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:45.367002351 +0000 UTC m=+120.599033171" watchObservedRunningTime="2026-09-17 17:36:45.367543722 +0000 UTC m=+120.599574540"
Sep 17 17:36:45.392279 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.392191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" podStartSLOduration=1.392170603 podStartE2EDuration="1.392170603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:45.391424062 +0000 UTC m=+120.623454881" watchObservedRunningTime="2026-09-17 17:36:45.392170603 +0000 UTC m=+120.624201427"
Sep 17 17:36:45.807792 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.807732    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:45.810937 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:45.810912    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:46.352636 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:46.352594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t" event={"ID":"26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3","Type":"ContainerStarted","Data":"6be831f1bc91d6d3c1605a924573cb137471e2e40344effad50566c1f3f313a9"}
Sep 17 17:36:46.352858 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:46.352838    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:46.354249 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:46.354226    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8559889cdb-wd9h9"
Sep 17 17:36:46.370385 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:46.370329    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-b9f8t" podStartSLOduration=1.401859229 podStartE2EDuration="2.370314051s" totalImagesPullingTime="968.454822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:46.369212816 +0000 UTC m=+121.601243640" watchObservedRunningTime="2026-09-17 17:36:46.370314051 +0000 UTC m=+121.602344869"
Sep 17 17:36:47.357547 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:47.357506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt" event={"ID":"68ec1a62-564e-403e-a5a3-85179a62f64f","Type":"ContainerStarted","Data":"21fba83974b64ab01930c0f435e3b1c0982df37e357422f843e20eb2a8b39f00"}
Sep 17 17:36:47.376649 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:47.376599    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xlltt" podStartSLOduration=33.600077667 podStartE2EDuration="35.376579381s" totalImagesPullingTime="1.776501714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:47.375855238 +0000 UTC m=+122.607886054" watchObservedRunningTime="2026-09-17 17:36:47.376579381 +0000 UTC m=+122.608610202"
Sep 17 17:36:47.485698 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:47.485669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:47.487947 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:47.487923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44e82ca8-6bf7-44a5-96aa-3c28ba930259-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dz2vf\" (UID: \"44e82ca8-6bf7-44a5-96aa-3c28ba930259\") " pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:47.713675 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:47.713613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xtzf6\""
Sep 17 17:36:47.721974 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:47.721957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dz2vf"
Sep 17 17:36:47.995433 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:47.995409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dz2vf"]
Sep 17 17:36:47.997987 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:47.997955    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e82ca8_6bf7_44a5_96aa_3c28ba930259.slice/crio-f62b59736764ffde019e41c66c08406f9f045646f3d04e4d7266a08f93304de9 WatchSource:0}: Error finding container f62b59736764ffde019e41c66c08406f9f045646f3d04e4d7266a08f93304de9: Status 404 returned error can't find the container with id f62b59736764ffde019e41c66c08406f9f045646f3d04e4d7266a08f93304de9
Sep 17 17:36:48.362126 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:48.362055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dz2vf" event={"ID":"44e82ca8-6bf7-44a5-96aa-3c28ba930259","Type":"ContainerStarted","Data":"3f6d580799772d3d747462cd0828da58765832636fa53512f4215ff60552e816"}
Sep 17 17:36:48.362126 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:48.362114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dz2vf" event={"ID":"44e82ca8-6bf7-44a5-96aa-3c28ba930259","Type":"ContainerStarted","Data":"f62b59736764ffde019e41c66c08406f9f045646f3d04e4d7266a08f93304de9"}
Sep 17 17:36:49.367837 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:49.367801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dz2vf" event={"ID":"44e82ca8-6bf7-44a5-96aa-3c28ba930259","Type":"ContainerStarted","Data":"bf0a3c381fc037820ab61ed7aef7903e9ec315bd888452343a7a2320538422fa"}
Sep 17 17:36:50.323270 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.323213    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"]
Sep 17 17:36:50.325313 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.325299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.328758 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.328734    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qspnk\""
Sep 17 17:36:50.328845 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.328775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:36:50.330099 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.330082    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:36:50.330906 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.330887    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:36:50.338610 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.338589    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"]
Sep 17 17:36:50.373240 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.373217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dz2vf" event={"ID":"44e82ca8-6bf7-44a5-96aa-3c28ba930259","Type":"ContainerStarted","Data":"c273f043061eceb4d9c5686715d2a371054baea71d6759c6fe5b1e6bf5a11fd1"}
Sep 17 17:36:50.400157 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.400120    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dz2vf" podStartSLOduration=33.475745061 podStartE2EDuration="35.400105164s" totalImagesPullingTime="1.924360103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:36:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:50.399179774 +0000 UTC m=+125.631210594" watchObservedRunningTime="2026-09-17 17:36:50.400105164 +0000 UTC m=+125.632136004"
Sep 17 17:36:50.408525 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.408504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chf7d\" (UniqueName: \"kubernetes.io/projected/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-kube-api-access-chf7d\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.408627 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.408544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.408627 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.408605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.408721 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.408658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.509662 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.509639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chf7d\" (UniqueName: \"kubernetes.io/projected/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-kube-api-access-chf7d\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.509767 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.509675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.509829 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.509811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.509872 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.509858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.510378 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.510356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.512152 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.512131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.512236 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.512218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.519970 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.519947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chf7d\" (UniqueName: \"kubernetes.io/projected/72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf-kube-api-access-chf7d\") pod \"prometheus-operator-7bd5d56d6-gwsxs\" (UID: \"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.633947 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.633899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"
Sep 17 17:36:50.904731 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:50.904628    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs"]
Sep 17 17:36:50.907431 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:50.907398    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bf23c8_91d6_4e7f_bd8e_94ea1acf8cbf.slice/crio-f252d64520f1cd4cd1540b6905b0eb4be1708501b3c72bf61b404618a0e87b16 WatchSource:0}: Error finding container f252d64520f1cd4cd1540b6905b0eb4be1708501b3c72bf61b404618a0e87b16: Status 404 returned error can't find the container with id f252d64520f1cd4cd1540b6905b0eb4be1708501b3c72bf61b404618a0e87b16
Sep 17 17:36:51.377769 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:51.377736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs" event={"ID":"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf","Type":"ContainerStarted","Data":"f252d64520f1cd4cd1540b6905b0eb4be1708501b3c72bf61b404618a0e87b16"}
Sep 17 17:36:52.383381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:52.383286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs" event={"ID":"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf","Type":"ContainerStarted","Data":"3e67f69b4cd789e841fbc56f9997534bddccc0912ed5b25405f91aa7df6629b0"}
Sep 17 17:36:52.383381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:52.383356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs" event={"ID":"72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf","Type":"ContainerStarted","Data":"d84b6cf2086bb9d6fb3654977864aa6807e74938ce04c4aa4abea7cc74cd2c21"}
Sep 17 17:36:54.430532 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.430502    2581 patch_prober.go:28] interesting pod/image-registry-54f86dc978-xcqrf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:36:54.430908 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.430553    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" podUID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:36:54.739031 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.738933    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gwsxs" podStartSLOduration=3.519507696 podStartE2EDuration="4.738912837s" totalImagesPullingTime="1.219405141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:52.402408114 +0000 UTC m=+127.634438932" watchObservedRunningTime="2026-09-17 17:36:54.738912837 +0000 UTC m=+129.970943658"
Sep 17 17:36:54.740095 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.740071    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b2pr8"]
Sep 17 17:36:54.742515 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.742497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.744939 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.744914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4zqpp\""
Sep 17 17:36:54.744939 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.744917    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:36:54.745205 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.745182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:36:54.745326 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.745264    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:36:54.841268 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-sys\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.841354 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czjl4\" (UniqueName: \"kubernetes.io/projected/90919883-fc9e-41e9-b4b0-14e7e8b961e8-kube-api-access-czjl4\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.841354 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-textfile\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.841354 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.841445 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.841445 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-root\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.841510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90919883-fc9e-41e9-b4b0-14e7e8b961e8-metrics-client-ca\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.841542 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-tls\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.841542 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.841533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-wtmp\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.941988 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.941965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-tls\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942126 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-wtmp\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942126 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-sys\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942126 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czjl4\" (UniqueName: \"kubernetes.io/projected/90919883-fc9e-41e9-b4b0-14e7e8b961e8-kube-api-access-czjl4\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942126 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-textfile\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-sys\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-root\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-wtmp\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90919883-fc9e-41e9-b4b0-14e7e8b961e8-root\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942572 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90919883-fc9e-41e9-b4b0-14e7e8b961e8-metrics-client-ca\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942572 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-textfile\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942803 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90919883-fc9e-41e9-b4b0-14e7e8b961e8-metrics-client-ca\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.942900 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.942876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.944296 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.944272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-tls\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.944375 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.944308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90919883-fc9e-41e9-b4b0-14e7e8b961e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:54.951674 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:54.951653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czjl4\" (UniqueName: \"kubernetes.io/projected/90919883-fc9e-41e9-b4b0-14e7e8b961e8-kube-api-access-czjl4\") pod \"node-exporter-b2pr8\" (UID: \"90919883-fc9e-41e9-b4b0-14e7e8b961e8\") " pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:55.051791 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.051768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b2pr8"
Sep 17 17:36:55.062157 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:55.062135    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90919883_fc9e_41e9_b4b0_14e7e8b961e8.slice/crio-9989b3dc55461d14f496d2b4e08611a0a4fcb3057ef9f88faa255290c7b95178 WatchSource:0}: Error finding container 9989b3dc55461d14f496d2b4e08611a0a4fcb3057ef9f88faa255290c7b95178: Status 404 returned error can't find the container with id 9989b3dc55461d14f496d2b4e08611a0a4fcb3057ef9f88faa255290c7b95178
Sep 17 17:36:55.395144 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.395075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b2pr8" event={"ID":"90919883-fc9e-41e9-b4b0-14e7e8b961e8","Type":"ContainerStarted","Data":"9989b3dc55461d14f496d2b4e08611a0a4fcb3057ef9f88faa255290c7b95178"}
Sep 17 17:36:55.888922 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.888884    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:55.893650 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.893621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.896719 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.896416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:36:55.896719 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.896463    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:36:55.896719 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.896541    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:36:55.896951 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.896735    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:36:55.896951 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.896750    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:36:55.896951 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.896780    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hn6xh\""
Sep 17 17:36:55.896951 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.896865    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:36:55.897290 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.897163    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:36:55.897290 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.897168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:36:55.897290 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.897243    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:36:55.913095 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.913069    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:55.949472 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949727 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-config-volume\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949727 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949727 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-web-config\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949727 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b498\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-kube-api-access-9b498\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949874 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949874 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949874 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-config-out\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949983 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949983 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:55.949983 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:55.949967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050578 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050689 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-config-volume\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050689 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050689 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-web-config\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b498\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-kube-api-access-9b498\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-config-out\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.050920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.051241 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.051241 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.050978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.053224 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.052125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.053224 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.052247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.055015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.053620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.055015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.053734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-config-volume\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.055015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.053986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.055015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.054892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-config-out\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.055015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.054951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.055472 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.055418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-web-config\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.055718 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.055679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.055974 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.055951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.056195 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.056174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.057047 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.057028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.060761 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.060745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b498\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-kube-api-access-9b498\") pod \"alertmanager-main-0\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.206431 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.206406    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:36:56.400162 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.400135    2581 generic.go:356] "Generic (PLEG): container finished" podID="90919883-fc9e-41e9-b4b0-14e7e8b961e8" containerID="b5cd2de2e24e261a65f711aa389ab9f767a95c1e0b9a3f48e0959ae2567f9d83" exitCode=0
Sep 17 17:36:56.400291 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.400195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b2pr8" event={"ID":"90919883-fc9e-41e9-b4b0-14e7e8b961e8","Type":"ContainerDied","Data":"b5cd2de2e24e261a65f711aa389ab9f767a95c1e0b9a3f48e0959ae2567f9d83"}
Sep 17 17:36:56.493312 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.493283    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:36:56.495803 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:56.495777    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a8a660_5767_4ac7_9255_7e805b4b37df.slice/crio-354eb7abad064a03f1762f0f63fa416b8064ec344a7fc802dcbf32dd086877c7 WatchSource:0}: Error finding container 354eb7abad064a03f1762f0f63fa416b8064ec344a7fc802dcbf32dd086877c7: Status 404 returned error can't find the container with id 354eb7abad064a03f1762f0f63fa416b8064ec344a7fc802dcbf32dd086877c7
Sep 17 17:36:56.555585 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.555553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:36:56.557619 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.557603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b665814a-b3dd-4590-9dbf-63559f1f5681-metrics-certs\") pod \"network-metrics-daemon-kc6h8\" (UID: \"b665814a-b3dd-4590-9dbf-63559f1f5681\") " pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:36:56.707636 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.707574    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xc26q\""
Sep 17 17:36:56.715663 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.715645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kc6h8"
Sep 17 17:36:56.989547 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:56.989366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kc6h8"]
Sep 17 17:36:56.991836 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:36:56.991814    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb665814a_b3dd_4590_9dbf_63559f1f5681.slice/crio-5d3ac3a6ed2cc1020005f71d46e48e4c99b6e4e5a9f6ed8cbf294cbe270845ff WatchSource:0}: Error finding container 5d3ac3a6ed2cc1020005f71d46e48e4c99b6e4e5a9f6ed8cbf294cbe270845ff: Status 404 returned error can't find the container with id 5d3ac3a6ed2cc1020005f71d46e48e4c99b6e4e5a9f6ed8cbf294cbe270845ff
Sep 17 17:36:57.406457 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:57.406421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b2pr8" event={"ID":"90919883-fc9e-41e9-b4b0-14e7e8b961e8","Type":"ContainerStarted","Data":"64e1c79d2e9eda0ec2d3b42c4be690db4908a08d297b6558d80b0ee4951d2838"}
Sep 17 17:36:57.406626 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:57.406487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b2pr8" event={"ID":"90919883-fc9e-41e9-b4b0-14e7e8b961e8","Type":"ContainerStarted","Data":"15b425daa2db9f50bb646fbfef306044e075fe37662d0b96229dc75ce1f56a1e"}
Sep 17 17:36:57.407762 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:57.407736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kc6h8" event={"ID":"b665814a-b3dd-4590-9dbf-63559f1f5681","Type":"ContainerStarted","Data":"5d3ac3a6ed2cc1020005f71d46e48e4c99b6e4e5a9f6ed8cbf294cbe270845ff"}
Sep 17 17:36:57.409147 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:57.409123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerStarted","Data":"354eb7abad064a03f1762f0f63fa416b8064ec344a7fc802dcbf32dd086877c7"}
Sep 17 17:36:57.428069 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:57.428030    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b2pr8" podStartSLOduration=2.401060848 podStartE2EDuration="3.428016745s" totalImagesPullingTime="968.275202ms" totalInitContainerRuntime="58.680695ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:57.426907343 +0000 UTC m=+132.658938159" watchObservedRunningTime="2026-09-17 17:36:57.428016745 +0000 UTC m=+132.660047563"
Sep 17 17:36:58.414851 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:58.414817    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerID="4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931" exitCode=0
Sep 17 17:36:58.415235 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:58.414909    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerDied","Data":"4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931"}
Sep 17 17:36:59.421346 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.421283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kc6h8" event={"ID":"b665814a-b3dd-4590-9dbf-63559f1f5681","Type":"ContainerStarted","Data":"8fafb53179d2d508fa14963130219bfe842a0054728bc001df5603e59611cf9c"}
Sep 17 17:36:59.421771 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.421393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kc6h8" event={"ID":"b665814a-b3dd-4590-9dbf-63559f1f5681","Type":"ContainerStarted","Data":"8427e3c45fbf05bc2f2cdf5b337b860856212d9ce1546666b83ca653c76352d4"}
Sep 17 17:36:59.440147 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.440106    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kc6h8" podStartSLOduration=131.973240637 podStartE2EDuration="2m13.440088784s" totalImagesPullingTime="1.466848147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:59.437851693 +0000 UTC m=+134.669882514" watchObservedRunningTime="2026-09-17 17:36:59.440088784 +0000 UTC m=+134.672119962"
Sep 17 17:36:59.466407 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.466381    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"]
Sep 17 17:36:59.469489 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.469470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:36:59.472038 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.472017    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 17:36:59.472337 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.472305    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nn6wt\""
Sep 17 17:36:59.481124 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.481090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"]
Sep 17 17:36:59.584114 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.584087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5ck\" (UniqueName: \"kubernetes.io/projected/3e46f46e-b59b-483f-8647-3786c1430d0c-kube-api-access-jm5ck\") pod \"monitoring-plugin-5984459ff8-vn6st\" (UID: \"3e46f46e-b59b-483f-8647-3786c1430d0c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:36:59.584268 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.584124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e46f46e-b59b-483f-8647-3786c1430d0c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vn6st\" (UID: \"3e46f46e-b59b-483f-8647-3786c1430d0c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:36:59.685163 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.685090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5ck\" (UniqueName: \"kubernetes.io/projected/3e46f46e-b59b-483f-8647-3786c1430d0c-kube-api-access-jm5ck\") pod \"monitoring-plugin-5984459ff8-vn6st\" (UID: \"3e46f46e-b59b-483f-8647-3786c1430d0c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:36:59.685163 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.685133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e46f46e-b59b-483f-8647-3786c1430d0c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vn6st\" (UID: \"3e46f46e-b59b-483f-8647-3786c1430d0c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:36:59.687551 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.687527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e46f46e-b59b-483f-8647-3786c1430d0c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vn6st\" (UID: \"3e46f46e-b59b-483f-8647-3786c1430d0c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:36:59.694739 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.694693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5ck\" (UniqueName: \"kubernetes.io/projected/3e46f46e-b59b-483f-8647-3786c1430d0c-kube-api-access-jm5ck\") pod \"monitoring-plugin-5984459ff8-vn6st\" (UID: \"3e46f46e-b59b-483f-8647-3786c1430d0c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:36:59.780313 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:36:59.780288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:37:00.144631 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:00.144604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"]
Sep 17 17:37:00.146680 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:37:00.146656    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e46f46e_b59b_483f_8647_3786c1430d0c.slice/crio-1fc279829df07180ab6d37d3b012be595fec6d619626ff425b6ae814ca75240f WatchSource:0}: Error finding container 1fc279829df07180ab6d37d3b012be595fec6d619626ff425b6ae814ca75240f: Status 404 returned error can't find the container with id 1fc279829df07180ab6d37d3b012be595fec6d619626ff425b6ae814ca75240f
Sep 17 17:37:00.427692 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:00.427663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerStarted","Data":"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a"}
Sep 17 17:37:00.428060 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:00.427747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerStarted","Data":"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7"}
Sep 17 17:37:00.428060 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:00.427787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerStarted","Data":"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f"}
Sep 17 17:37:00.428060 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:00.427824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerStarted","Data":"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687"}
Sep 17 17:37:00.428060 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:00.427849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerStarted","Data":"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a"}
Sep 17 17:37:00.428926 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:00.428904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st" event={"ID":"3e46f46e-b59b-483f-8647-3786c1430d0c","Type":"ContainerStarted","Data":"1fc279829df07180ab6d37d3b012be595fec6d619626ff425b6ae814ca75240f"}
Sep 17 17:37:01.439532 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:01.439458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerStarted","Data":"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a"}
Sep 17 17:37:01.471818 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:01.471761    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.911556272 podStartE2EDuration="6.471742118s" totalImagesPullingTime="3.54536718s" totalInitContainerRuntime="14.818666ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:01.470334611 +0000 UTC m=+136.702365443" watchObservedRunningTime="2026-09-17 17:37:01.471742118 +0000 UTC m=+136.703772937"
Sep 17 17:37:02.445617 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:02.445574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st" event={"ID":"3e46f46e-b59b-483f-8647-3786c1430d0c","Type":"ContainerStarted","Data":"4d8af37007d61b2895e3aa2c765a1e0779d17407134989bbb80059e2a4cfce45"}
Sep 17 17:37:02.446078 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:02.445935    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:37:02.451019 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:02.450994    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st"
Sep 17 17:37:02.464628 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:02.464587    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vn6st" podStartSLOduration=1.823486882 podStartE2EDuration="3.464572662s" totalImagesPullingTime="1.64108578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:02.4636824 +0000 UTC m=+137.695713222" watchObservedRunningTime="2026-09-17 17:37:02.464572662 +0000 UTC m=+137.696603507"
Sep 17 17:37:04.430128 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:04.430093    2581 patch_prober.go:28] interesting pod/image-registry-54f86dc978-xcqrf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:04.430572 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:04.430152    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" podUID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:04.824037 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:04.824006    2581 patch_prober.go:28] interesting pod/image-registry-6b6df65cc4-jd6g9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:04.824199 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:04.824055    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" podUID="e7307cdc-4814-4e1a-803c-1aad13e3ddb1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:06.357715 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:06.357673    2581 patch_prober.go:28] interesting pod/image-registry-6b6df65cc4-jd6g9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:06.358046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:06.357745    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" podUID="e7307cdc-4814-4e1a-803c-1aad13e3ddb1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:08.795726 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:08.795675    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-cprrn"]
Sep 17 17:37:08.800249 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:08.800227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-cprrn"
Sep 17 17:37:08.804731 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:08.804466    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:37:08.804731 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:08.804468    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8tr7r\""
Sep 17 17:37:08.805135 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:08.805118    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:37:08.811338 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:08.811313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-cprrn"]
Sep 17 17:37:08.959887 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:08.959862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9lv\" (UniqueName: \"kubernetes.io/projected/4dbeed9c-f53c-4e22-afc0-a7549595158b-kube-api-access-ll9lv\") pod \"downloads-6594f4c75-cprrn\" (UID: \"4dbeed9c-f53c-4e22-afc0-a7549595158b\") " pod="openshift-console/downloads-6594f4c75-cprrn"
Sep 17 17:37:09.061188 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.061122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9lv\" (UniqueName: \"kubernetes.io/projected/4dbeed9c-f53c-4e22-afc0-a7549595158b-kube-api-access-ll9lv\") pod \"downloads-6594f4c75-cprrn\" (UID: \"4dbeed9c-f53c-4e22-afc0-a7549595158b\") " pod="openshift-console/downloads-6594f4c75-cprrn"
Sep 17 17:37:09.071648 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.071624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll9lv\" (UniqueName: \"kubernetes.io/projected/4dbeed9c-f53c-4e22-afc0-a7549595158b-kube-api-access-ll9lv\") pod \"downloads-6594f4c75-cprrn\" (UID: \"4dbeed9c-f53c-4e22-afc0-a7549595158b\") " pod="openshift-console/downloads-6594f4c75-cprrn"
Sep 17 17:37:09.110508 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.110484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-cprrn"
Sep 17 17:37:09.386825 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.386801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-cprrn"]
Sep 17 17:37:09.388826 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:37:09.388801    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbeed9c_f53c_4e22_afc0_a7549595158b.slice/crio-a6dcffb6c6385a567518c9171271f80b20bc1c463e9a96bf88c6577f4a3d9551 WatchSource:0}: Error finding container a6dcffb6c6385a567518c9171271f80b20bc1c463e9a96bf88c6577f4a3d9551: Status 404 returned error can't find the container with id a6dcffb6c6385a567518c9171271f80b20bc1c463e9a96bf88c6577f4a3d9551
Sep 17 17:37:09.444558 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.444505    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" podUID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" containerName="registry" containerID="cri-o://5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5" gracePeriod=30
Sep 17 17:37:09.473691 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.473663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-cprrn" event={"ID":"4dbeed9c-f53c-4e22-afc0-a7549595158b","Type":"ContainerStarted","Data":"a6dcffb6c6385a567518c9171271f80b20bc1c463e9a96bf88c6577f4a3d9551"}
Sep 17 17:37:09.672271 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.672249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:37:09.767905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.767876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qftmx\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-kube-api-access-qftmx\") pod \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") "
Sep 17 17:37:09.768022 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.767913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") pod \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") "
Sep 17 17:37:09.768022 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.767938    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-certificates\") pod \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") "
Sep 17 17:37:09.768022 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.767971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-trusted-ca\") pod \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") "
Sep 17 17:37:09.768022 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.768013    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c68274e4-1f0f-4801-b58b-a3af19bd5c91-ca-trust-extracted\") pod \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") "
Sep 17 17:37:09.768230 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.768110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-bound-sa-token\") pod \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") "
Sep 17 17:37:09.768230 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.768141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-image-registry-private-configuration\") pod \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") "
Sep 17 17:37:09.768230 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.768174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-installation-pull-secrets\") pod \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\" (UID: \"c68274e4-1f0f-4801-b58b-a3af19bd5c91\") "
Sep 17 17:37:09.768423 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.768366    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c68274e4-1f0f-4801-b58b-a3af19bd5c91" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:09.768543 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.768431    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c68274e4-1f0f-4801-b58b-a3af19bd5c91" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:09.770246 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.770217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-kube-api-access-qftmx" (OuterVolumeSpecName: "kube-api-access-qftmx") pod "c68274e4-1f0f-4801-b58b-a3af19bd5c91" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91"). InnerVolumeSpecName "kube-api-access-qftmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:09.770446 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.770425    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c68274e4-1f0f-4801-b58b-a3af19bd5c91" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:09.770600 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.770571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c68274e4-1f0f-4801-b58b-a3af19bd5c91" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:09.770687 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.770649    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c68274e4-1f0f-4801-b58b-a3af19bd5c91" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:09.770782 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.770756    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c68274e4-1f0f-4801-b58b-a3af19bd5c91" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:09.776356 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.776333    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68274e4-1f0f-4801-b58b-a3af19bd5c91-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c68274e4-1f0f-4801-b58b-a3af19bd5c91" (UID: "c68274e4-1f0f-4801-b58b-a3af19bd5c91"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:09.869354 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.869335    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c68274e4-1f0f-4801-b58b-a3af19bd5c91-ca-trust-extracted\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:09.869354 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.869356    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-bound-sa-token\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:09.869683 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.869365    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-image-registry-private-configuration\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:09.869683 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.869374    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c68274e4-1f0f-4801-b58b-a3af19bd5c91-installation-pull-secrets\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:09.869683 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.869382    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qftmx\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-kube-api-access-qftmx\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:09.869683 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.869392    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:09.869683 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.869407    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-registry-certificates\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:09.869683 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:09.869419    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c68274e4-1f0f-4801-b58b-a3af19bd5c91-trusted-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:10.479663 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.479577    2581 generic.go:356] "Generic (PLEG): container finished" podID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" containerID="5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5" exitCode=0
Sep 17 17:37:10.479663 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.479619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf"
Sep 17 17:37:10.479663 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.479649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" event={"ID":"c68274e4-1f0f-4801-b58b-a3af19bd5c91","Type":"ContainerDied","Data":"5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5"}
Sep 17 17:37:10.479915 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.479728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54f86dc978-xcqrf" event={"ID":"c68274e4-1f0f-4801-b58b-a3af19bd5c91","Type":"ContainerDied","Data":"86d263f9c789700f6c8177d23357ef2fe01a3111eab55cbd2c61e5db0029e837"}
Sep 17 17:37:10.479915 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.479760    2581 scope.go:117] "RemoveContainer" containerID="5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5"
Sep 17 17:37:10.489820 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.489800    2581 scope.go:117] "RemoveContainer" containerID="5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5"
Sep 17 17:37:10.490046 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:37:10.490025    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5\": container with ID starting with 5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5 not found: ID does not exist" containerID="5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5"
Sep 17 17:37:10.490149 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.490053    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5"} err="failed to get container status \"5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5\": rpc error: code = NotFound desc = could not find container \"5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5\": container with ID starting with 5a32e797776941ffdb27c7ee7f044dcfd03d0de74ebb05fcd21ed8c5cec008e5 not found: ID does not exist"
Sep 17 17:37:10.510266 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.508517    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54f86dc978-xcqrf"]
Sep 17 17:37:10.513796 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.513774    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54f86dc978-xcqrf"]
Sep 17 17:37:10.898867 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:10.898838    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" path="/var/lib/kubelet/pods/c68274e4-1f0f-4801-b58b-a3af19bd5c91/volumes"
Sep 17 17:37:13.917342 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.917310    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5558b8c8b9-9z574"]
Sep 17 17:37:13.917804 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.917748    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" containerName="registry"
Sep 17 17:37:13.917804 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.917765    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" containerName="registry"
Sep 17 17:37:13.917921 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.917847    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c68274e4-1f0f-4801-b58b-a3af19bd5c91" containerName="registry"
Sep 17 17:37:13.922152 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.922134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:13.927120 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.927094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:37:13.927799 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.927781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:37:13.928391 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.928346    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:37:13.928391 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.928378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4kmn8\""
Sep 17 17:37:13.929008 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.928984    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:37:13.931984 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.931483    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:37:13.933843 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:13.933797    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5558b8c8b9-9z574"]
Sep 17 17:37:14.001722 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.001687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-service-ca\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.001829 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.001768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-serving-cert\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.001829 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.001807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-oauth-serving-cert\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.001829 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.001827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-oauth-config\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.001979 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.001854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxfn\" (UniqueName: \"kubernetes.io/projected/f76ce99d-810e-49e4-8b34-a8417ee74b48-kube-api-access-kwxfn\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.001979 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.001887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-config\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.102408 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.102380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-oauth-serving-cert\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.102526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.102416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-oauth-config\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.102526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.102431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxfn\" (UniqueName: \"kubernetes.io/projected/f76ce99d-810e-49e4-8b34-a8417ee74b48-kube-api-access-kwxfn\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.102526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.102461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-config\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.102526 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.102505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-service-ca\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.102755 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.102566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-serving-cert\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.103210 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.103182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-oauth-serving-cert\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.103445 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.103420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-config\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.103445 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.103438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-service-ca\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.105165 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.105144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-serving-cert\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.105241 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.105185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-oauth-config\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.113508 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.113485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxfn\" (UniqueName: \"kubernetes.io/projected/f76ce99d-810e-49e4-8b34-a8417ee74b48-kube-api-access-kwxfn\") pod \"console-5558b8c8b9-9z574\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") " pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.233614 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.233545    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:14.544521 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.544497    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5558b8c8b9-9z574"]
Sep 17 17:37:14.546760 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:37:14.546728    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76ce99d_810e_49e4_8b34_a8417ee74b48.slice/crio-2013955ca956439ed533b75aac271b8aa04b998790974b8115ecd8bc44f0dd65 WatchSource:0}: Error finding container 2013955ca956439ed533b75aac271b8aa04b998790974b8115ecd8bc44f0dd65: Status 404 returned error can't find the container with id 2013955ca956439ed533b75aac271b8aa04b998790974b8115ecd8bc44f0dd65
Sep 17 17:37:14.825454 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.825363    2581 patch_prober.go:28] interesting pod/image-registry-6b6df65cc4-jd6g9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:14.825454 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:14.825416    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" podUID="e7307cdc-4814-4e1a-803c-1aad13e3ddb1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:15.506417 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:15.506350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5558b8c8b9-9z574" event={"ID":"f76ce99d-810e-49e4-8b34-a8417ee74b48","Type":"ContainerStarted","Data":"2013955ca956439ed533b75aac271b8aa04b998790974b8115ecd8bc44f0dd65"}
Sep 17 17:37:16.360294 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:16.359895    2581 patch_prober.go:28] interesting pod/image-registry-6b6df65cc4-jd6g9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:16.360294 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:16.359954    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" podUID="e7307cdc-4814-4e1a-803c-1aad13e3ddb1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:19.526951 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:19.526915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5558b8c8b9-9z574" event={"ID":"f76ce99d-810e-49e4-8b34-a8417ee74b48","Type":"ContainerStarted","Data":"fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8"}
Sep 17 17:37:19.550956 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:19.550912    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5558b8c8b9-9z574" podStartSLOduration=2.031518381 podStartE2EDuration="6.550897161s" totalImagesPullingTime="4.51937878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:19.549627499 +0000 UTC m=+154.781658319" watchObservedRunningTime="2026-09-17 17:37:19.550897161 +0000 UTC m=+154.782927982"
Sep 17 17:37:23.275418 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.275381    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fbfff5c58-n9c6g"]
Sep 17 17:37:23.279684 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.279664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.288394 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.288176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:37:23.289797 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.289777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbfff5c58-n9c6g"]
Sep 17 17:37:23.380460 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.380433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-trusted-ca-bundle\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.380605 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.380472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2l8\" (UniqueName: \"kubernetes.io/projected/d3e83e82-e832-40b5-9189-18b024c510b2-kube-api-access-zk2l8\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.380605 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.380505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-serving-cert\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.380605 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.380532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-console-config\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.380605 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.380597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-oauth-serving-cert\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.380811 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.380639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-oauth-config\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.380811 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.380683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-service-ca\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.481749 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.481694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-oauth-config\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.481879 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.481753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-service-ca\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.481879 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.481805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-trusted-ca-bundle\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.481879 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.481822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2l8\" (UniqueName: \"kubernetes.io/projected/d3e83e82-e832-40b5-9189-18b024c510b2-kube-api-access-zk2l8\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.481879 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.481851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-serving-cert\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.481879 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.481878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-console-config\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.482146 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.482037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-oauth-serving-cert\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.482653 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.482632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-console-config\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.482769 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.482632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-service-ca\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.482817 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.482783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-oauth-serving-cert\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.482965 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.482922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-trusted-ca-bundle\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.484486 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.484438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-oauth-config\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.484593 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.484572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-serving-cert\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.491688 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.491660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2l8\" (UniqueName: \"kubernetes.io/projected/d3e83e82-e832-40b5-9189-18b024c510b2-kube-api-access-zk2l8\") pod \"console-5fbfff5c58-n9c6g\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") " pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:23.591825 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:23.591763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:24.209689 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:37:24.209649    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zmpdc" podUID="ecbf5bec-f51d-44ff-ba63-a47812c3b733"
Sep 17 17:37:24.233762 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.233736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:24.233915 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.233823    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:24.239201 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.239179    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:24.548239 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.548211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:37:24.552576 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.552553    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:24.825805 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.825728    2581 patch_prober.go:28] interesting pod/image-registry-6b6df65cc4-jd6g9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:24.825805 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.825778    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" podUID="e7307cdc-4814-4e1a-803c-1aad13e3ddb1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:24.826015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.825827    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:37:24.826339 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.826316    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"31e18894ec4a7945077a5a7bfd33827a364f8e1ca78fd083bd04c9a94fcb33a5"} pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 17 17:37:24.830822 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.830797    2581 patch_prober.go:28] interesting pod/image-registry-6b6df65cc4-jd6g9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:24.830918 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:24.830840    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" podUID="e7307cdc-4814-4e1a-803c-1aad13e3ddb1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:26.558209 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:26.558164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-cprrn" event={"ID":"4dbeed9c-f53c-4e22-afc0-a7549595158b","Type":"ContainerStarted","Data":"df9a9f43dcc75d43a17e4dfd4faafe756d762eab00e9994bfdd1ad6f1a82a2c8"}
Sep 17 17:37:26.579599 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:26.579560    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-cprrn" podStartSLOduration=1.521763228 podStartE2EDuration="18.579544531s" totalImagesPullingTime="17.057781303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:26.578985634 +0000 UTC m=+161.811016453" watchObservedRunningTime="2026-09-17 17:37:26.579544531 +0000 UTC m=+161.811575365"
Sep 17 17:37:26.622596 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:26.622571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbfff5c58-n9c6g"]
Sep 17 17:37:26.625325 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:37:26.625297    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e83e82_e832_40b5_9189_18b024c510b2.slice/crio-ad1bc36c9f83c8d1d64d2bddbbb74616715139ea449464c1cd983baab1fb58a0 WatchSource:0}: Error finding container ad1bc36c9f83c8d1d64d2bddbbb74616715139ea449464c1cd983baab1fb58a0: Status 404 returned error can't find the container with id ad1bc36c9f83c8d1d64d2bddbbb74616715139ea449464c1cd983baab1fb58a0
Sep 17 17:37:27.565206 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:27.565169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbfff5c58-n9c6g" event={"ID":"d3e83e82-e832-40b5-9189-18b024c510b2","Type":"ContainerStarted","Data":"3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419"}
Sep 17 17:37:27.565657 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:27.565225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbfff5c58-n9c6g" event={"ID":"d3e83e82-e832-40b5-9189-18b024c510b2","Type":"ContainerStarted","Data":"ad1bc36c9f83c8d1d64d2bddbbb74616715139ea449464c1cd983baab1fb58a0"}
Sep 17 17:37:27.567169 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:27.567139    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b7c5595-dde6-4720-bb6d-7633122dc737" containerID="4647fd8b58b21fe7c0a8ba00c7b503b66a229ced4f7bd7b6c96a00c70c3502ed" exitCode=0
Sep 17 17:37:27.567292 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:27.567170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q" event={"ID":"1b7c5595-dde6-4720-bb6d-7633122dc737","Type":"ContainerDied","Data":"4647fd8b58b21fe7c0a8ba00c7b503b66a229ced4f7bd7b6c96a00c70c3502ed"}
Sep 17 17:37:27.567473 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:27.567451    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-cprrn"
Sep 17 17:37:27.567635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:27.567615    2581 scope.go:117] "RemoveContainer" containerID="4647fd8b58b21fe7c0a8ba00c7b503b66a229ced4f7bd7b6c96a00c70c3502ed"
Sep 17 17:37:27.579744 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:27.579722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-cprrn"
Sep 17 17:37:27.663127 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:27.663067    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fbfff5c58-n9c6g" podStartSLOduration=4.663044041 podStartE2EDuration="4.663044041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:27.629429629 +0000 UTC m=+162.861460449" watchObservedRunningTime="2026-09-17 17:37:27.663044041 +0000 UTC m=+162.895074862"
Sep 17 17:37:28.574745 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:28.574684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-jp46q" event={"ID":"1b7c5595-dde6-4720-bb6d-7633122dc737","Type":"ContainerStarted","Data":"5f8e1b2fef734f4d80f989d98c88b744ca624d6d892e2005d7a4052998929896"}
Sep 17 17:37:29.038492 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:29.038457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:37:29.041067 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:29.041041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbf5bec-f51d-44ff-ba63-a47812c3b733-metrics-tls\") pod \"dns-default-zmpdc\" (UID: \"ecbf5bec-f51d-44ff-ba63-a47812c3b733\") " pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:37:29.051657 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:29.051634    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gbrs5\""
Sep 17 17:37:29.059955 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:29.059931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:37:29.357642 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:29.357569    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmpdc"]
Sep 17 17:37:29.360864 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:37:29.360840    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbf5bec_f51d_44ff_ba63_a47812c3b733.slice/crio-27d4e9df9065345432c48ca23164ef0cc78497716727b501e9d959b9587ce924 WatchSource:0}: Error finding container 27d4e9df9065345432c48ca23164ef0cc78497716727b501e9d959b9587ce924: Status 404 returned error can't find the container with id 27d4e9df9065345432c48ca23164ef0cc78497716727b501e9d959b9587ce924
Sep 17 17:37:29.580560 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:29.580522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmpdc" event={"ID":"ecbf5bec-f51d-44ff-ba63-a47812c3b733","Type":"ContainerStarted","Data":"27d4e9df9065345432c48ca23164ef0cc78497716727b501e9d959b9587ce924"}
Sep 17 17:37:32.597075 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:32.597033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmpdc" event={"ID":"ecbf5bec-f51d-44ff-ba63-a47812c3b733","Type":"ContainerStarted","Data":"ab62f29aeec0101db94c2ce9ea60323f037b3dc0e06f7c49e4ca5c7a133560fa"}
Sep 17 17:37:32.597568 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:32.597106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:37:32.597568 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:32.597132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmpdc" event={"ID":"ecbf5bec-f51d-44ff-ba63-a47812c3b733","Type":"ContainerStarted","Data":"3beebd32fc19c872da2c1f11f33dcb4a4c2495aa63e78ee046d6af7c8e4e1b73"}
Sep 17 17:37:32.618060 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:32.617830    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zmpdc" podStartSLOduration=129.524175866 podStartE2EDuration="2m11.617811235s" totalImagesPullingTime="2.093635369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:32.617333513 +0000 UTC m=+167.849364333" watchObservedRunningTime="2026-09-17 17:37:32.617811235 +0000 UTC m=+167.849842055"
Sep 17 17:37:33.591937 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:33.591894    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:33.592226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:33.592195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:33.597553 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:33.597530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:33.611005 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:33.610981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:37:33.693786 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:33.693759    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5558b8c8b9-9z574"]
Sep 17 17:37:34.830808 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:34.830775    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:37:40.640980 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:40.640948    2581 generic.go:356] "Generic (PLEG): container finished" podID="51a506c7-09c6-4921-91b2-fd8ea4a044c8" containerID="02fdb661d0095a9db3cee7b758bb5e56786ef18d4ef454f0186814169fa10bf5" exitCode=0
Sep 17 17:37:40.641349 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:40.641020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h" event={"ID":"51a506c7-09c6-4921-91b2-fd8ea4a044c8","Type":"ContainerDied","Data":"02fdb661d0095a9db3cee7b758bb5e56786ef18d4ef454f0186814169fa10bf5"}
Sep 17 17:37:40.641392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:40.641349    2581 scope.go:117] "RemoveContainer" containerID="02fdb661d0095a9db3cee7b758bb5e56786ef18d4ef454f0186814169fa10bf5"
Sep 17 17:37:41.188086 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:41.188056    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-685d7d68b5-xzmx9_e197166b-a6d8-4e4b-acee-184344294481/cluster-samples-operator/0.log"
Sep 17 17:37:41.196085 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:41.196064    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-685d7d68b5-xzmx9_e197166b-a6d8-4e4b-acee-184344294481/cluster-samples-operator-watch/0.log"
Sep 17 17:37:41.647109 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:41.647079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-dfd2h" event={"ID":"51a506c7-09c6-4921-91b2-fd8ea4a044c8","Type":"ContainerStarted","Data":"1de1776a8d0dbffbf35948f9f09ab20665c32f0726b7ec67747562f8fbcc514e"}
Sep 17 17:37:42.608333 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:42.608306    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zmpdc"
Sep 17 17:37:49.846925 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:49.846884    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" podUID="e7307cdc-4814-4e1a-803c-1aad13e3ddb1" containerName="registry" containerID="cri-o://31e18894ec4a7945077a5a7bfd33827a364f8e1ca78fd083bd04c9a94fcb33a5" gracePeriod=30
Sep 17 17:37:51.691066 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:51.691030    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7307cdc-4814-4e1a-803c-1aad13e3ddb1" containerID="31e18894ec4a7945077a5a7bfd33827a364f8e1ca78fd083bd04c9a94fcb33a5" exitCode=0
Sep 17 17:37:51.691533 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:51.691156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" event={"ID":"e7307cdc-4814-4e1a-803c-1aad13e3ddb1","Type":"ContainerDied","Data":"31e18894ec4a7945077a5a7bfd33827a364f8e1ca78fd083bd04c9a94fcb33a5"}
Sep 17 17:37:51.691533 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:51.691205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9" event={"ID":"e7307cdc-4814-4e1a-803c-1aad13e3ddb1","Type":"ContainerStarted","Data":"3e1fa9b662a1f9d5e4e581265ab66dc2d4a3d6305c2dc0d31f849cc303f5950f"}
Sep 17 17:37:51.691533 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:51.691252    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:37:58.717512 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:58.717464    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5558b8c8b9-9z574" podUID="f76ce99d-810e-49e4-8b34-a8417ee74b48" containerName="console" containerID="cri-o://fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8" gracePeriod=15
Sep 17 17:37:58.988140 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:58.988074    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:59.055961 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.055935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwxfn\" (UniqueName: \"kubernetes.io/projected/f76ce99d-810e-49e4-8b34-a8417ee74b48-kube-api-access-kwxfn\") pod \"f76ce99d-810e-49e4-8b34-a8417ee74b48\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") "
Sep 17 17:37:59.056123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.055968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-serving-cert\") pod \"f76ce99d-810e-49e4-8b34-a8417ee74b48\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") "
Sep 17 17:37:59.056123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.055999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-service-ca\") pod \"f76ce99d-810e-49e4-8b34-a8417ee74b48\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") "
Sep 17 17:37:59.056123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.056029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-oauth-config\") pod \"f76ce99d-810e-49e4-8b34-a8417ee74b48\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") "
Sep 17 17:37:59.056123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.056064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-oauth-serving-cert\") pod \"f76ce99d-810e-49e4-8b34-a8417ee74b48\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") "
Sep 17 17:37:59.056123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.056083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-config\") pod \"f76ce99d-810e-49e4-8b34-a8417ee74b48\" (UID: \"f76ce99d-810e-49e4-8b34-a8417ee74b48\") "
Sep 17 17:37:59.056911 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.056875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f76ce99d-810e-49e4-8b34-a8417ee74b48" (UID: "f76ce99d-810e-49e4-8b34-a8417ee74b48"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:59.056911 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.056891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-service-ca" (OuterVolumeSpecName: "service-ca") pod "f76ce99d-810e-49e4-8b34-a8417ee74b48" (UID: "f76ce99d-810e-49e4-8b34-a8417ee74b48"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:59.057089 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.056926    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-config" (OuterVolumeSpecName: "console-config") pod "f76ce99d-810e-49e4-8b34-a8417ee74b48" (UID: "f76ce99d-810e-49e4-8b34-a8417ee74b48"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:59.061956 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.061925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76ce99d-810e-49e4-8b34-a8417ee74b48-kube-api-access-kwxfn" (OuterVolumeSpecName: "kube-api-access-kwxfn") pod "f76ce99d-810e-49e4-8b34-a8417ee74b48" (UID: "f76ce99d-810e-49e4-8b34-a8417ee74b48"). InnerVolumeSpecName "kube-api-access-kwxfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:59.063285 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.063260    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f76ce99d-810e-49e4-8b34-a8417ee74b48" (UID: "f76ce99d-810e-49e4-8b34-a8417ee74b48"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:59.063421 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.063401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f76ce99d-810e-49e4-8b34-a8417ee74b48" (UID: "f76ce99d-810e-49e4-8b34-a8417ee74b48"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:59.156697 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.156668    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-service-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:59.156697 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.156694    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-oauth-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:59.156697 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.156720    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-oauth-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:59.156896 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.156729    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:59.156896 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.156737    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwxfn\" (UniqueName: \"kubernetes.io/projected/f76ce99d-810e-49e4-8b34-a8417ee74b48-kube-api-access-kwxfn\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:59.156896 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.156745    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76ce99d-810e-49e4-8b34-a8417ee74b48-console-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:37:59.728793 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.728759    2581 generic.go:356] "Generic (PLEG): container finished" podID="f76ce99d-810e-49e4-8b34-a8417ee74b48" containerID="fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8" exitCode=0
Sep 17 17:37:59.729204 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.728821    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5558b8c8b9-9z574"
Sep 17 17:37:59.729204 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.728843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5558b8c8b9-9z574" event={"ID":"f76ce99d-810e-49e4-8b34-a8417ee74b48","Type":"ContainerDied","Data":"fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8"}
Sep 17 17:37:59.729204 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.728916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5558b8c8b9-9z574" event={"ID":"f76ce99d-810e-49e4-8b34-a8417ee74b48","Type":"ContainerDied","Data":"2013955ca956439ed533b75aac271b8aa04b998790974b8115ecd8bc44f0dd65"}
Sep 17 17:37:59.729204 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.728936    2581 scope.go:117] "RemoveContainer" containerID="fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8"
Sep 17 17:37:59.737311 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.737289    2581 scope.go:117] "RemoveContainer" containerID="fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8"
Sep 17 17:37:59.737563 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:37:59.737544    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8\": container with ID starting with fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8 not found: ID does not exist" containerID="fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8"
Sep 17 17:37:59.737619 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.737572    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8"} err="failed to get container status \"fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8\": rpc error: code = NotFound desc = could not find container \"fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8\": container with ID starting with fa4379b9ade130b34ebf6343e6d704ce087de815d548885a3ef02128172682f8 not found: ID does not exist"
Sep 17 17:37:59.752997 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.752976    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5558b8c8b9-9z574"]
Sep 17 17:37:59.758617 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:37:59.758597    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5558b8c8b9-9z574"]
Sep 17 17:38:00.898249 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:00.898217    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f76ce99d-810e-49e4-8b34-a8417ee74b48" path="/var/lib/kubelet/pods/f76ce99d-810e-49e4-8b34-a8417ee74b48/volumes"
Sep 17 17:38:12.699393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:12.699364    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b6df65cc4-jd6g9"
Sep 17 17:38:15.225223 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:15.225168    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:15.796865 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:15.796799    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="alertmanager" containerID="cri-o://cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a" gracePeriod=120
Sep 17 17:38:15.797082 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:15.796889    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy-metric" containerID="cri-o://bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a" gracePeriod=120
Sep 17 17:38:15.797082 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:15.796898    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy" containerID="cri-o://9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7" gracePeriod=120
Sep 17 17:38:15.797082 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:15.796948    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="prom-label-proxy" containerID="cri-o://f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a" gracePeriod=120
Sep 17 17:38:15.797082 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:15.796879    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy-web" containerID="cri-o://3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f" gracePeriod=120
Sep 17 17:38:15.797082 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:15.796898    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="config-reloader" containerID="cri-o://1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687" gracePeriod=120
Sep 17 17:38:16.805730 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:16.805675    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerID="f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a" exitCode=0
Sep 17 17:38:16.805730 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:16.805728    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerID="9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7" exitCode=0
Sep 17 17:38:16.805730 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:16.805738    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerID="1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687" exitCode=0
Sep 17 17:38:16.806189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:16.805746    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerID="cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a" exitCode=0
Sep 17 17:38:16.806189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:16.805737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerDied","Data":"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a"}
Sep 17 17:38:16.806189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:16.805818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerDied","Data":"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7"}
Sep 17 17:38:16.806189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:16.805848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerDied","Data":"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687"}
Sep 17 17:38:16.806189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:16.805874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerDied","Data":"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a"}
Sep 17 17:38:17.029893 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.029872    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:17.101108 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101045    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-cluster-tls-config\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-web-config\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101167    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-tls-assets\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101428 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101234    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-web\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101428 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b498\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-kube-api-access-9b498\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101428 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-main-db\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101428 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-config-volume\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101428 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-metrics-client-ca\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101428 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-trusted-ca-bundle\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101428 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-main-tls\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101803 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-config-out\") pod \"95a8a660-5767-4ac7-9255-7e805b4b37df\" (UID: \"95a8a660-5767-4ac7-9255-7e805b4b37df\") "
Sep 17 17:38:17.101803 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.101747    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:38:17.102452 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.102367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:17.102452 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.102386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:17.103920 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.103879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:17.104030 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.103952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-config-out" (OuterVolumeSpecName: "config-out") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:38:17.105429 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.105401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:17.105527 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.105424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:17.105527 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.105514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:17.106600 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.106573    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-kube-api-access-9b498" (OuterVolumeSpecName: "kube-api-access-9b498") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "kube-api-access-9b498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:17.106939 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.106901    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-config-volume" (OuterVolumeSpecName: "config-volume") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:17.107405 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.107004    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:17.109934 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.109681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:17.116417 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.116397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-web-config" (OuterVolumeSpecName: "web-config") pod "95a8a660-5767-4ac7-9255-7e805b4b37df" (UID: "95a8a660-5767-4ac7-9255-7e805b4b37df"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:17.203003 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.202981    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203003 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203003    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b498\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-kube-api-access-9b498\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203012    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-main-db\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203021    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-config-volume\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203030    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-metrics-client-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203038    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95a8a660-5767-4ac7-9255-7e805b4b37df-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203046    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-main-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203055    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95a8a660-5767-4ac7-9255-7e805b4b37df-config-out\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203062    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-cluster-tls-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203073    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203082    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-web-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203090    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95a8a660-5767-4ac7-9255-7e805b4b37df-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.203123 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.203098    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95a8a660-5767-4ac7-9255-7e805b4b37df-tls-assets\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:17.817534 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.817496    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerID="bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a" exitCode=0
Sep 17 17:38:17.817534 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.817524    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerID="3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f" exitCode=0
Sep 17 17:38:17.817967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.817574    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:17.817967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.817582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerDied","Data":"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a"}
Sep 17 17:38:17.817967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.817674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerDied","Data":"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f"}
Sep 17 17:38:17.817967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.817717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"95a8a660-5767-4ac7-9255-7e805b4b37df","Type":"ContainerDied","Data":"354eb7abad064a03f1762f0f63fa416b8064ec344a7fc802dcbf32dd086877c7"}
Sep 17 17:38:17.817967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.817728    2581 scope.go:117] "RemoveContainer" containerID="f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a"
Sep 17 17:38:17.825402 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.825384    2581 scope.go:117] "RemoveContainer" containerID="bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a"
Sep 17 17:38:17.832253 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.832236    2581 scope.go:117] "RemoveContainer" containerID="9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7"
Sep 17 17:38:17.838685 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.838669    2581 scope.go:117] "RemoveContainer" containerID="3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f"
Sep 17 17:38:17.845554 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.845531    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:17.845900 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.845883    2581 scope.go:117] "RemoveContainer" containerID="1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687"
Sep 17 17:38:17.851126 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.851107    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:17.855028 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.855010    2581 scope.go:117] "RemoveContainer" containerID="cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a"
Sep 17 17:38:17.861600 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.861583    2581 scope.go:117] "RemoveContainer" containerID="4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931"
Sep 17 17:38:17.869476 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.869459    2581 scope.go:117] "RemoveContainer" containerID="f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a"
Sep 17 17:38:17.869990 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:38:17.869969    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a\": container with ID starting with f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a not found: ID does not exist" containerID="f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a"
Sep 17 17:38:17.870069 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.870000    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a"} err="failed to get container status \"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a\": rpc error: code = NotFound desc = could not find container \"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a\": container with ID starting with f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a not found: ID does not exist"
Sep 17 17:38:17.870069 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.870024    2581 scope.go:117] "RemoveContainer" containerID="bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a"
Sep 17 17:38:17.870368 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:38:17.870346    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a\": container with ID starting with bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a not found: ID does not exist" containerID="bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a"
Sep 17 17:38:17.870411 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.870376    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a"} err="failed to get container status \"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a\": rpc error: code = NotFound desc = could not find container \"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a\": container with ID starting with bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a not found: ID does not exist"
Sep 17 17:38:17.870411 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.870397    2581 scope.go:117] "RemoveContainer" containerID="9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7"
Sep 17 17:38:17.870637 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:38:17.870619    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7\": container with ID starting with 9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7 not found: ID does not exist" containerID="9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7"
Sep 17 17:38:17.870684 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.870641    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7"} err="failed to get container status \"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7\": rpc error: code = NotFound desc = could not find container \"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7\": container with ID starting with 9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7 not found: ID does not exist"
Sep 17 17:38:17.870684 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.870656    2581 scope.go:117] "RemoveContainer" containerID="3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f"
Sep 17 17:38:17.870921 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:38:17.870905    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f\": container with ID starting with 3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f not found: ID does not exist" containerID="3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f"
Sep 17 17:38:17.871017 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.870928    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f"} err="failed to get container status \"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f\": rpc error: code = NotFound desc = could not find container \"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f\": container with ID starting with 3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f not found: ID does not exist"
Sep 17 17:38:17.871017 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.870949    2581 scope.go:117] "RemoveContainer" containerID="1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687"
Sep 17 17:38:17.871225 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:38:17.871209    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687\": container with ID starting with 1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687 not found: ID does not exist" containerID="1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687"
Sep 17 17:38:17.871259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.871229    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687"} err="failed to get container status \"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687\": rpc error: code = NotFound desc = could not find container \"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687\": container with ID starting with 1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687 not found: ID does not exist"
Sep 17 17:38:17.871259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.871242    2581 scope.go:117] "RemoveContainer" containerID="cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a"
Sep 17 17:38:17.871461 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:38:17.871443    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a\": container with ID starting with cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a not found: ID does not exist" containerID="cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a"
Sep 17 17:38:17.871502 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.871466    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a"} err="failed to get container status \"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a\": rpc error: code = NotFound desc = could not find container \"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a\": container with ID starting with cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a not found: ID does not exist"
Sep 17 17:38:17.871502 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.871479    2581 scope.go:117] "RemoveContainer" containerID="4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931"
Sep 17 17:38:17.871715 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:38:17.871691    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931\": container with ID starting with 4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931 not found: ID does not exist" containerID="4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931"
Sep 17 17:38:17.871755 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.871724    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931"} err="failed to get container status \"4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931\": rpc error: code = NotFound desc = could not find container \"4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931\": container with ID starting with 4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931 not found: ID does not exist"
Sep 17 17:38:17.871755 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.871736    2581 scope.go:117] "RemoveContainer" containerID="f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a"
Sep 17 17:38:17.871939 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.871922    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a"} err="failed to get container status \"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a\": rpc error: code = NotFound desc = could not find container \"f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a\": container with ID starting with f4ec73503ca161a38c061fa384784447523a7f92198e922b13f4da7d9080f44a not found: ID does not exist"
Sep 17 17:38:17.871976 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.871942    2581 scope.go:117] "RemoveContainer" containerID="bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a"
Sep 17 17:38:17.872177 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.872159    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a"} err="failed to get container status \"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a\": rpc error: code = NotFound desc = could not find container \"bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a\": container with ID starting with bf6bef74237c0228604750c25cb090c1d6d3bbaa4f60366f239ee6643d43e74a not found: ID does not exist"
Sep 17 17:38:17.872222 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.872178    2581 scope.go:117] "RemoveContainer" containerID="9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7"
Sep 17 17:38:17.872403 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.872389    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7"} err="failed to get container status \"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7\": rpc error: code = NotFound desc = could not find container \"9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7\": container with ID starting with 9c85c4027b87fa453ec52c5ca521944422fb4e16edd74fdd0eaefbaa89802df7 not found: ID does not exist"
Sep 17 17:38:17.872444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.872404    2581 scope.go:117] "RemoveContainer" containerID="3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f"
Sep 17 17:38:17.872623 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.872605    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f"} err="failed to get container status \"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f\": rpc error: code = NotFound desc = could not find container \"3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f\": container with ID starting with 3d4bc6479c5c0ddf3028f181968b729cc8b1b636b837b13ea4d88cceb0407d0f not found: ID does not exist"
Sep 17 17:38:17.872671 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.872624    2581 scope.go:117] "RemoveContainer" containerID="1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687"
Sep 17 17:38:17.872868 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.872851    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687"} err="failed to get container status \"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687\": rpc error: code = NotFound desc = could not find container \"1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687\": container with ID starting with 1bf2997a51122d183946a08a0ed384f2e913652ec94140c1136e47e2788b0687 not found: ID does not exist"
Sep 17 17:38:17.872912 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.872868    2581 scope.go:117] "RemoveContainer" containerID="cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a"
Sep 17 17:38:17.873083 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.873068    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a"} err="failed to get container status \"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a\": rpc error: code = NotFound desc = could not find container \"cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a\": container with ID starting with cf0c60f99b4207211df0c328b78918997024c9e63992a619483bfe9ecc34485a not found: ID does not exist"
Sep 17 17:38:17.873119 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.873084    2581 scope.go:117] "RemoveContainer" containerID="4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931"
Sep 17 17:38:17.873292 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.873276    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931"} err="failed to get container status \"4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931\": rpc error: code = NotFound desc = could not find container \"4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931\": container with ID starting with 4717f9a46730d9bfe7931d8dbd8f201812b2ec9d012d29bcbd8b2ea645d78931 not found: ID does not exist"
Sep 17 17:38:17.884864 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.884841    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:17.885099 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885087    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy-web"
Sep 17 17:38:17.885141 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885101    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy-web"
Sep 17 17:38:17.885141 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885111    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="init-config-reloader"
Sep 17 17:38:17.885141 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885117    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="init-config-reloader"
Sep 17 17:38:17.885141 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885123    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="alertmanager"
Sep 17 17:38:17.885141 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885129    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="alertmanager"
Sep 17 17:38:17.885141 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885135    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="config-reloader"
Sep 17 17:38:17.885141 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885140    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="config-reloader"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885147    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885152    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885163    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="prom-label-proxy"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885168    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="prom-label-proxy"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885178    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy-metric"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885183    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy-metric"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885193    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76ce99d-810e-49e4-8b34-a8417ee74b48" containerName="console"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885198    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76ce99d-810e-49e4-8b34-a8417ee74b48" containerName="console"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885243    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="prom-label-proxy"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885251    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="config-reloader"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy-metric"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885263    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="alertmanager"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885269    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy-web"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885274    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f76ce99d-810e-49e4-8b34-a8417ee74b48" containerName="console"
Sep 17 17:38:17.885340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.885279    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" containerName="kube-rbac-proxy"
Sep 17 17:38:17.888999 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.888985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:17.892991 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.892352    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hn6xh\""
Sep 17 17:38:17.893101 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.893034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:38:17.893279 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.893117    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:38:17.893413 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.893401    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:38:17.893479 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.893418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:38:17.893540 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.893128    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:38:17.893540 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.893424    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:38:17.893540 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.893506    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:38:17.893736 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.893630    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:38:17.898818 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.898799    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:38:17.908987 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:17.908964    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:18.008276 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbw54\" (UniqueName: \"kubernetes.io/projected/6bc3705c-084c-4c36-a694-382e2ae9eeb3-kube-api-access-pbw54\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008370 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008370 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc3705c-084c-4c36-a694-382e2ae9eeb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008370 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008489 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008489 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bc3705c-084c-4c36-a694-382e2ae9eeb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008489 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bc3705c-084c-4c36-a694-382e2ae9eeb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008579 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008579 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008579 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-web-config\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008662 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bc3705c-084c-4c36-a694-382e2ae9eeb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008662 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bc3705c-084c-4c36-a694-382e2ae9eeb3-config-out\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.008662 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.008658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.109955 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.109897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc3705c-084c-4c36-a694-382e2ae9eeb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.109955 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.109936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.109962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bc3705c-084c-4c36-a694-382e2ae9eeb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bc3705c-084c-4c36-a694-382e2ae9eeb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-web-config\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bc3705c-084c-4c36-a694-382e2ae9eeb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110541 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bc3705c-084c-4c36-a694-382e2ae9eeb3-config-out\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110541 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110541 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbw54\" (UniqueName: \"kubernetes.io/projected/6bc3705c-084c-4c36-a694-382e2ae9eeb3-kube-api-access-pbw54\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110541 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.110541 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.110448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bc3705c-084c-4c36-a694-382e2ae9eeb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.111374 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.111028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc3705c-084c-4c36-a694-382e2ae9eeb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.111374 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.111353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bc3705c-084c-4c36-a694-382e2ae9eeb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.113415 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.113098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.113415 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.113344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bc3705c-084c-4c36-a694-382e2ae9eeb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.113558 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.113437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bc3705c-084c-4c36-a694-382e2ae9eeb3-config-out\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.113558 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.113444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.113658 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.113629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.113841 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.113814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.113937 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.113914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.114030 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.114015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.115270 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.115253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bc3705c-084c-4c36-a694-382e2ae9eeb3-web-config\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.119965 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.119946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbw54\" (UniqueName: \"kubernetes.io/projected/6bc3705c-084c-4c36-a694-382e2ae9eeb3-kube-api-access-pbw54\") pod \"alertmanager-main-0\" (UID: \"6bc3705c-084c-4c36-a694-382e2ae9eeb3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.199787 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.199768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:18.482989 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.482967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:18.484799 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:38:18.484761    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc3705c_084c_4c36_a694_382e2ae9eeb3.slice/crio-9a2bf522459ffa7b515c5c06d7f62330f621b8c5a721e292d96b3af9b34e9b9c WatchSource:0}: Error finding container 9a2bf522459ffa7b515c5c06d7f62330f621b8c5a721e292d96b3af9b34e9b9c: Status 404 returned error can't find the container with id 9a2bf522459ffa7b515c5c06d7f62330f621b8c5a721e292d96b3af9b34e9b9c
Sep 17 17:38:18.824349 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.824318    2581 generic.go:356] "Generic (PLEG): container finished" podID="6bc3705c-084c-4c36-a694-382e2ae9eeb3" containerID="6a23d8e96faae368fd75ab04145e6a621ef20f9bb6b0a49b3e3e544bb39cc76a" exitCode=0
Sep 17 17:38:18.824666 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.824389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bc3705c-084c-4c36-a694-382e2ae9eeb3","Type":"ContainerDied","Data":"6a23d8e96faae368fd75ab04145e6a621ef20f9bb6b0a49b3e3e544bb39cc76a"}
Sep 17 17:38:18.824666 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.824448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bc3705c-084c-4c36-a694-382e2ae9eeb3","Type":"ContainerStarted","Data":"9a2bf522459ffa7b515c5c06d7f62330f621b8c5a721e292d96b3af9b34e9b9c"}
Sep 17 17:38:18.899043 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:18.899015    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a8a660-5767-4ac7-9255-7e805b4b37df" path="/var/lib/kubelet/pods/95a8a660-5767-4ac7-9255-7e805b4b37df/volumes"
Sep 17 17:38:19.299832 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.299799    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"]
Sep 17 17:38:19.302097 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.302080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.309465 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.308929    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 17:38:19.309465 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.309161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 17:38:19.309465 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.309292    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 17:38:19.310329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.310074    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 17:38:19.310329 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.310216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-l6x4p\""
Sep 17 17:38:19.315314 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.310885    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 17:38:19.320992 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.320476    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 17:38:19.323559 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.323533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"]
Sep 17 17:38:19.421015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.420936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-telemeter-client-tls\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.421142 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.421030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428km\" (UniqueName: \"kubernetes.io/projected/722a1816-b9f5-4efe-b5ee-a614e6422524-kube-api-access-428km\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.421142 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.421089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-serving-certs-ca-bundle\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.421142 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.421107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-secret-telemeter-client\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.421142 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.421121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-metrics-client-ca\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.421292 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.421194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-federate-client-tls\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.421292 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.421221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.421292 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.421243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.521608 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.521577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-federate-client-tls\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.521608 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.521610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.521795 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.521630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.521795 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.521758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-telemeter-client-tls\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.521860 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.521839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428km\" (UniqueName: \"kubernetes.io/projected/722a1816-b9f5-4efe-b5ee-a614e6422524-kube-api-access-428km\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.521909 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.521897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-serving-certs-ca-bundle\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.521946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.521916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-secret-telemeter-client\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.521946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.521933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-metrics-client-ca\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.522598 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.522572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-serving-certs-ca-bundle\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.522598 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.522583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-metrics-client-ca\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.522757 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.522665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/722a1816-b9f5-4efe-b5ee-a614e6422524-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.524317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.524298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-secret-telemeter-client\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.524460 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.524438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-telemeter-client-tls\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.524531 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.524517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.524568 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.524527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/722a1816-b9f5-4efe-b5ee-a614e6422524-federate-client-tls\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.531666 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.531645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-428km\" (UniqueName: \"kubernetes.io/projected/722a1816-b9f5-4efe-b5ee-a614e6422524-kube-api-access-428km\") pod \"telemeter-client-6f4dcbbfb9-685zg\" (UID: \"722a1816-b9f5-4efe-b5ee-a614e6422524\") " pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.616666 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.616638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"
Sep 17 17:38:19.832515 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.832475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bc3705c-084c-4c36-a694-382e2ae9eeb3","Type":"ContainerStarted","Data":"1949cd3c0cf927d73c35855a2880cb05744b43184719f65b5444128093bedbc1"}
Sep 17 17:38:19.832946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.832560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bc3705c-084c-4c36-a694-382e2ae9eeb3","Type":"ContainerStarted","Data":"4bcc29e7cae0f7dba128cb15eb9c26c095611e27a9acf4b2c1d41123858337c5"}
Sep 17 17:38:19.832946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.832601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bc3705c-084c-4c36-a694-382e2ae9eeb3","Type":"ContainerStarted","Data":"aea251d6e965d0563fcb2fcff3925f7c459c94cbdf270a5ce3bc278076ecfca9"}
Sep 17 17:38:19.832946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.832639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bc3705c-084c-4c36-a694-382e2ae9eeb3","Type":"ContainerStarted","Data":"fbd67dc2128aca4ad725058ef2b0a3d1d23199e471391a7211cb267b9b7b001c"}
Sep 17 17:38:19.832946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.832676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bc3705c-084c-4c36-a694-382e2ae9eeb3","Type":"ContainerStarted","Data":"2fe8aef7bda4dfcf5920643907ea00881bcc658c7b30255591eac82208d9456c"}
Sep 17 17:38:19.833366 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.833264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bc3705c-084c-4c36-a694-382e2ae9eeb3","Type":"ContainerStarted","Data":"fad67bc7fab09169fac33483009deeda299f0298d60468517d30b2465cab5afd"}
Sep 17 17:38:19.864577 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.864534    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.854399031 podStartE2EDuration="2.864519774s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.120743ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:19.862250795 +0000 UTC m=+215.094281627" watchObservedRunningTime="2026-09-17 17:38:19.864519774 +0000 UTC m=+215.096550593"
Sep 17 17:38:19.901760 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:19.901557    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg"]
Sep 17 17:38:19.905176 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:38:19.905142    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722a1816_b9f5_4efe_b5ee_a614e6422524.slice/crio-92eb577b5c0b2aa219b75ca9b4ef14fac2820ce88bc5a2f0f4347b1b49af7b83 WatchSource:0}: Error finding container 92eb577b5c0b2aa219b75ca9b4ef14fac2820ce88bc5a2f0f4347b1b49af7b83: Status 404 returned error can't find the container with id 92eb577b5c0b2aa219b75ca9b4ef14fac2820ce88bc5a2f0f4347b1b49af7b83
Sep 17 17:38:20.837074 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:20.837039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg" event={"ID":"722a1816-b9f5-4efe-b5ee-a614e6422524","Type":"ContainerStarted","Data":"92eb577b5c0b2aa219b75ca9b4ef14fac2820ce88bc5a2f0f4347b1b49af7b83"}
Sep 17 17:38:21.843549 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:21.843479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg" event={"ID":"722a1816-b9f5-4efe-b5ee-a614e6422524","Type":"ContainerStarted","Data":"065890dcbec60bf73f2689ddb7922d16726845444d40830958b22bd5f8d8a613"}
Sep 17 17:38:21.843549 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:21.843541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg" event={"ID":"722a1816-b9f5-4efe-b5ee-a614e6422524","Type":"ContainerStarted","Data":"ab551183a398173171778edb0fd4bad80ca6515439dbbb0a1a98c67381d4367b"}
Sep 17 17:38:21.843906 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:21.843570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg" event={"ID":"722a1816-b9f5-4efe-b5ee-a614e6422524","Type":"ContainerStarted","Data":"8ba1a063e3859883e1a9c79402a9409cfe83df08cc56711c7fdbe534064b7b4d"}
Sep 17 17:38:21.869002 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:21.868957    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f4dcbbfb9-685zg" podStartSLOduration=1.19338096 podStartE2EDuration="2.868940989s" totalImagesPullingTime="1.675560029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:21.867574945 +0000 UTC m=+217.099605764" watchObservedRunningTime="2026-09-17 17:38:21.868940989 +0000 UTC m=+217.100971787"
Sep 17 17:38:21.964140 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:21.964116    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-769cc8496-h2pnq"]
Sep 17 17:38:21.971265 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:21.971239    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:21.980603 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:21.980577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-769cc8496-h2pnq"]
Sep 17 17:38:22.042189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.042168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-serving-cert\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.042300 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.042197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-oauth-serving-cert\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.042300 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.042225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-service-ca\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.042369 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.042300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9jh\" (UniqueName: \"kubernetes.io/projected/b196eac0-fc37-401d-8d07-70411e4e3d1f-kube-api-access-jr9jh\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.042369 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.042335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-config\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.042369 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.042358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-oauth-config\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.042460 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.042377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-trusted-ca-bundle\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.143485 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.143433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-service-ca\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.143485 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.143469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9jh\" (UniqueName: \"kubernetes.io/projected/b196eac0-fc37-401d-8d07-70411e4e3d1f-kube-api-access-jr9jh\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.143608 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.143491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-config\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.143608 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.143510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-oauth-config\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.143608 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.143547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-trusted-ca-bundle\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.143608 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.143580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-serving-cert\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.143608 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.143600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-oauth-serving-cert\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.144365 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.144347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-oauth-serving-cert\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.144441 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.144390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-service-ca\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.144441 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.144391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-config\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.144510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.144480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-trusted-ca-bundle\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.145850 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.145833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-oauth-config\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.146019 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.146002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-serving-cert\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.154753 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.154728    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9jh\" (UniqueName: \"kubernetes.io/projected/b196eac0-fc37-401d-8d07-70411e4e3d1f-kube-api-access-jr9jh\") pod \"console-769cc8496-h2pnq\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") " pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.280925 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.280679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:22.567559 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.567405    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-769cc8496-h2pnq"]
Sep 17 17:38:22.569699 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:38:22.569674    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb196eac0_fc37_401d_8d07_70411e4e3d1f.slice/crio-ed981a52581808d01efd41f60cf790651cf86c1306043bc25029a7013f6f9d52 WatchSource:0}: Error finding container ed981a52581808d01efd41f60cf790651cf86c1306043bc25029a7013f6f9d52: Status 404 returned error can't find the container with id ed981a52581808d01efd41f60cf790651cf86c1306043bc25029a7013f6f9d52
Sep 17 17:38:22.850347 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.850256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769cc8496-h2pnq" event={"ID":"b196eac0-fc37-401d-8d07-70411e4e3d1f","Type":"ContainerStarted","Data":"132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981"}
Sep 17 17:38:22.850347 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.850309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769cc8496-h2pnq" event={"ID":"b196eac0-fc37-401d-8d07-70411e4e3d1f","Type":"ContainerStarted","Data":"ed981a52581808d01efd41f60cf790651cf86c1306043bc25029a7013f6f9d52"}
Sep 17 17:38:22.884946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:22.884897    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-769cc8496-h2pnq" podStartSLOduration=1.884882083 podStartE2EDuration="1.884882083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:22.884085089 +0000 UTC m=+218.116115922" watchObservedRunningTime="2026-09-17 17:38:22.884882083 +0000 UTC m=+218.116912901"
Sep 17 17:38:32.281742 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:32.281693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:32.282181 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:32.281796    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:32.286498 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:32.286477    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:32.897496 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:32.897470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:38:32.955390 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:32.955354    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fbfff5c58-n9c6g"]
Sep 17 17:38:57.977187 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:57.977073    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fbfff5c58-n9c6g" podUID="d3e83e82-e832-40b5-9189-18b024c510b2" containerName="console" containerID="cri-o://3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419" gracePeriod=15
Sep 17 17:38:58.224301 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.224280    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:38:58.323331 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-serving-cert\") pod \"d3e83e82-e832-40b5-9189-18b024c510b2\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") "
Sep 17 17:38:58.323445 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323339    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk2l8\" (UniqueName: \"kubernetes.io/projected/d3e83e82-e832-40b5-9189-18b024c510b2-kube-api-access-zk2l8\") pod \"d3e83e82-e832-40b5-9189-18b024c510b2\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") "
Sep 17 17:38:58.323445 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-trusted-ca-bundle\") pod \"d3e83e82-e832-40b5-9189-18b024c510b2\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") "
Sep 17 17:38:58.323445 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-console-config\") pod \"d3e83e82-e832-40b5-9189-18b024c510b2\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") "
Sep 17 17:38:58.323567 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-oauth-config\") pod \"d3e83e82-e832-40b5-9189-18b024c510b2\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") "
Sep 17 17:38:58.323567 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-oauth-serving-cert\") pod \"d3e83e82-e832-40b5-9189-18b024c510b2\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") "
Sep 17 17:38:58.323567 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-service-ca\") pod \"d3e83e82-e832-40b5-9189-18b024c510b2\" (UID: \"d3e83e82-e832-40b5-9189-18b024c510b2\") "
Sep 17 17:38:58.323912 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d3e83e82-e832-40b5-9189-18b024c510b2" (UID: "d3e83e82-e832-40b5-9189-18b024c510b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:58.324008 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d3e83e82-e832-40b5-9189-18b024c510b2" (UID: "d3e83e82-e832-40b5-9189-18b024c510b2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:58.324008 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.323887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-console-config" (OuterVolumeSpecName: "console-config") pod "d3e83e82-e832-40b5-9189-18b024c510b2" (UID: "d3e83e82-e832-40b5-9189-18b024c510b2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:58.324127 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.324072    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-service-ca" (OuterVolumeSpecName: "service-ca") pod "d3e83e82-e832-40b5-9189-18b024c510b2" (UID: "d3e83e82-e832-40b5-9189-18b024c510b2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:58.325514 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.325493    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d3e83e82-e832-40b5-9189-18b024c510b2" (UID: "d3e83e82-e832-40b5-9189-18b024c510b2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:58.325669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.325647    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d3e83e82-e832-40b5-9189-18b024c510b2" (UID: "d3e83e82-e832-40b5-9189-18b024c510b2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:58.325876 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.325859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e83e82-e832-40b5-9189-18b024c510b2-kube-api-access-zk2l8" (OuterVolumeSpecName: "kube-api-access-zk2l8") pod "d3e83e82-e832-40b5-9189-18b024c510b2" (UID: "d3e83e82-e832-40b5-9189-18b024c510b2"). InnerVolumeSpecName "kube-api-access-zk2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:58.424506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.424479    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-console-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:58.424506 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.424502    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-oauth-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:58.424645 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.424517    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-oauth-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:58.424645 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.424531    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-service-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:58.424645 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.424543    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3e83e82-e832-40b5-9189-18b024c510b2-console-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:58.424645 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.424556    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk2l8\" (UniqueName: \"kubernetes.io/projected/d3e83e82-e832-40b5-9189-18b024c510b2-kube-api-access-zk2l8\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:58.424645 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:58.424568    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3e83e82-e832-40b5-9189-18b024c510b2-trusted-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:38:59.002835 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.002805    2581 generic.go:356] "Generic (PLEG): container finished" podID="d3e83e82-e832-40b5-9189-18b024c510b2" containerID="3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419" exitCode=0
Sep 17 17:38:59.003205 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.002841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbfff5c58-n9c6g" event={"ID":"d3e83e82-e832-40b5-9189-18b024c510b2","Type":"ContainerDied","Data":"3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419"}
Sep 17 17:38:59.003205 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.002855    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbfff5c58-n9c6g"
Sep 17 17:38:59.003205 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.002873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbfff5c58-n9c6g" event={"ID":"d3e83e82-e832-40b5-9189-18b024c510b2","Type":"ContainerDied","Data":"ad1bc36c9f83c8d1d64d2bddbbb74616715139ea449464c1cd983baab1fb58a0"}
Sep 17 17:38:59.003205 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.002895    2581 scope.go:117] "RemoveContainer" containerID="3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419"
Sep 17 17:38:59.012045 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.012022    2581 scope.go:117] "RemoveContainer" containerID="3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419"
Sep 17 17:38:59.012373 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:38:59.012349    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419\": container with ID starting with 3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419 not found: ID does not exist" containerID="3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419"
Sep 17 17:38:59.012447 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.012387    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419"} err="failed to get container status \"3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419\": rpc error: code = NotFound desc = could not find container \"3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419\": container with ID starting with 3e620165c948f1aa9a10623c3f55b2ed7d23425cb91f6003838c03652c8d8419 not found: ID does not exist"
Sep 17 17:38:59.036614 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.036587    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fbfff5c58-n9c6g"]
Sep 17 17:38:59.040804 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:38:59.040782    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fbfff5c58-n9c6g"]
Sep 17 17:39:00.899131 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:00.899088    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3e83e82-e832-40b5-9189-18b024c510b2" path="/var/lib/kubelet/pods/d3e83e82-e832-40b5-9189-18b024c510b2/volumes"
Sep 17 17:39:43.635294 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.635264    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b98ffbbf9-nj6ht"]
Sep 17 17:39:43.635751 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.635571    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e83e82-e832-40b5-9189-18b024c510b2" containerName="console"
Sep 17 17:39:43.635751 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.635583    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e83e82-e832-40b5-9189-18b024c510b2" containerName="console"
Sep 17 17:39:43.635751 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.635633    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e83e82-e832-40b5-9189-18b024c510b2" containerName="console"
Sep 17 17:39:43.638349 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.638330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.651779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.651750    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b98ffbbf9-nj6ht"]
Sep 17 17:39:43.752914 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.752886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-serving-cert\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.753046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.752923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-oauth-serving-cert\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.753046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.752946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-trusted-ca-bundle\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.753046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.752961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-config\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.753145 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.753044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-oauth-config\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.753145 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.753077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-service-ca\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.753145 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.753118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzj7\" (UniqueName: \"kubernetes.io/projected/41f0a15b-1f10-4d44-a64d-4a34a126c074-kube-api-access-kmzj7\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.853647 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.853608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-oauth-config\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.853647 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.853652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-service-ca\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.853858 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.853685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzj7\" (UniqueName: \"kubernetes.io/projected/41f0a15b-1f10-4d44-a64d-4a34a126c074-kube-api-access-kmzj7\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.853858 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.853744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-serving-cert\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.853858 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.853787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-oauth-serving-cert\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.853858 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.853819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-trusted-ca-bundle\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.853858 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.853842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-config\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.854532 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.854508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-config\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.854655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.854507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-service-ca\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.854655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.854557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-oauth-serving-cert\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.854799 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.854675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-trusted-ca-bundle\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.856194 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.856167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-serving-cert\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.856194 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.856189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-oauth-config\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.863475 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.863457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzj7\" (UniqueName: \"kubernetes.io/projected/41f0a15b-1f10-4d44-a64d-4a34a126c074-kube-api-access-kmzj7\") pod \"console-6b98ffbbf9-nj6ht\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") " pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:43.947626 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:43.947576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:44.219970 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:44.219945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b98ffbbf9-nj6ht"]
Sep 17 17:39:44.222434 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:39:44.222403    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f0a15b_1f10_4d44_a64d_4a34a126c074.slice/crio-01d326c0eeb563e91784fc4de07bccc168e4b5d4822d3c9d0da47cf83f84c58f WatchSource:0}: Error finding container 01d326c0eeb563e91784fc4de07bccc168e4b5d4822d3c9d0da47cf83f84c58f: Status 404 returned error can't find the container with id 01d326c0eeb563e91784fc4de07bccc168e4b5d4822d3c9d0da47cf83f84c58f
Sep 17 17:39:45.187840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:45.187806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b98ffbbf9-nj6ht" event={"ID":"41f0a15b-1f10-4d44-a64d-4a34a126c074","Type":"ContainerStarted","Data":"ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f"}
Sep 17 17:39:45.188305 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:45.187864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b98ffbbf9-nj6ht" event={"ID":"41f0a15b-1f10-4d44-a64d-4a34a126c074","Type":"ContainerStarted","Data":"01d326c0eeb563e91784fc4de07bccc168e4b5d4822d3c9d0da47cf83f84c58f"}
Sep 17 17:39:45.209274 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:45.209225    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b98ffbbf9-nj6ht" podStartSLOduration=2.209209852 podStartE2EDuration="2.209209852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:45.207435501 +0000 UTC m=+300.439466320" watchObservedRunningTime="2026-09-17 17:39:45.209209852 +0000 UTC m=+300.441240671"
Sep 17 17:39:46.741599 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:46.741573    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:39:46.748412 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:46.748393    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:39:46.751355 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:46.751333    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:39:53.948113 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:53.948076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:53.950623 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:53.948170    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:53.953039 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:53.953018    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:54.229388 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:54.229313    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:39:54.328607 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:39:54.328575    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-769cc8496-h2pnq"]
Sep 17 17:40:19.348497 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.348450    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-769cc8496-h2pnq" podUID="b196eac0-fc37-401d-8d07-70411e4e3d1f" containerName="console" containerID="cri-o://132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981" gracePeriod=15
Sep 17 17:40:19.591407 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.591384    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:40:19.648012 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.647948    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-serving-cert\") pod \"b196eac0-fc37-401d-8d07-70411e4e3d1f\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") "
Sep 17 17:40:19.648012 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.647985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-oauth-config\") pod \"b196eac0-fc37-401d-8d07-70411e4e3d1f\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") "
Sep 17 17:40:19.648193 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648027    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-service-ca\") pod \"b196eac0-fc37-401d-8d07-70411e4e3d1f\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") "
Sep 17 17:40:19.648193 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-oauth-serving-cert\") pod \"b196eac0-fc37-401d-8d07-70411e4e3d1f\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") "
Sep 17 17:40:19.648193 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648074    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-trusted-ca-bundle\") pod \"b196eac0-fc37-401d-8d07-70411e4e3d1f\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") "
Sep 17 17:40:19.648193 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648096    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr9jh\" (UniqueName: \"kubernetes.io/projected/b196eac0-fc37-401d-8d07-70411e4e3d1f-kube-api-access-jr9jh\") pod \"b196eac0-fc37-401d-8d07-70411e4e3d1f\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") "
Sep 17 17:40:19.648193 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648129    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-config\") pod \"b196eac0-fc37-401d-8d07-70411e4e3d1f\" (UID: \"b196eac0-fc37-401d-8d07-70411e4e3d1f\") "
Sep 17 17:40:19.648551 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b196eac0-fc37-401d-8d07-70411e4e3d1f" (UID: "b196eac0-fc37-401d-8d07-70411e4e3d1f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:19.648551 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-service-ca" (OuterVolumeSpecName: "service-ca") pod "b196eac0-fc37-401d-8d07-70411e4e3d1f" (UID: "b196eac0-fc37-401d-8d07-70411e4e3d1f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:19.648764 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648590    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-config" (OuterVolumeSpecName: "console-config") pod "b196eac0-fc37-401d-8d07-70411e4e3d1f" (UID: "b196eac0-fc37-401d-8d07-70411e4e3d1f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:19.648764 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.648603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b196eac0-fc37-401d-8d07-70411e4e3d1f" (UID: "b196eac0-fc37-401d-8d07-70411e4e3d1f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:19.650096 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.650074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b196eac0-fc37-401d-8d07-70411e4e3d1f" (UID: "b196eac0-fc37-401d-8d07-70411e4e3d1f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:40:19.650204 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.650182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b196eac0-fc37-401d-8d07-70411e4e3d1f" (UID: "b196eac0-fc37-401d-8d07-70411e4e3d1f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:40:19.650402 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.650380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b196eac0-fc37-401d-8d07-70411e4e3d1f-kube-api-access-jr9jh" (OuterVolumeSpecName: "kube-api-access-jr9jh") pod "b196eac0-fc37-401d-8d07-70411e4e3d1f" (UID: "b196eac0-fc37-401d-8d07-70411e4e3d1f"). InnerVolumeSpecName "kube-api-access-jr9jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:40:19.749182 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.749152    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-service-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:40:19.749182 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.749178    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-oauth-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:40:19.749361 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.749188    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-trusted-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:40:19.749361 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.749199    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr9jh\" (UniqueName: \"kubernetes.io/projected/b196eac0-fc37-401d-8d07-70411e4e3d1f-kube-api-access-jr9jh\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:40:19.749361 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.749209    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:40:19.749361 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.749217    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:40:19.749361 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:19.749225    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b196eac0-fc37-401d-8d07-70411e4e3d1f-console-oauth-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:40:20.336422 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.336385    2581 generic.go:356] "Generic (PLEG): container finished" podID="b196eac0-fc37-401d-8d07-70411e4e3d1f" containerID="132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981" exitCode=0
Sep 17 17:40:20.336594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.336448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769cc8496-h2pnq"
Sep 17 17:40:20.336594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.336473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769cc8496-h2pnq" event={"ID":"b196eac0-fc37-401d-8d07-70411e4e3d1f","Type":"ContainerDied","Data":"132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981"}
Sep 17 17:40:20.336594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.336546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769cc8496-h2pnq" event={"ID":"b196eac0-fc37-401d-8d07-70411e4e3d1f","Type":"ContainerDied","Data":"ed981a52581808d01efd41f60cf790651cf86c1306043bc25029a7013f6f9d52"}
Sep 17 17:40:20.336594 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.336567    2581 scope.go:117] "RemoveContainer" containerID="132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981"
Sep 17 17:40:20.346216 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.346200    2581 scope.go:117] "RemoveContainer" containerID="132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981"
Sep 17 17:40:20.346492 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:40:20.346473    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981\": container with ID starting with 132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981 not found: ID does not exist" containerID="132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981"
Sep 17 17:40:20.346562 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.346503    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981"} err="failed to get container status \"132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981\": rpc error: code = NotFound desc = could not find container \"132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981\": container with ID starting with 132b34fb05ee8801a2f6a547d1a99d0d2adffcdb1add0fbbc3356a6082c25981 not found: ID does not exist"
Sep 17 17:40:20.367826 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.367804    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-769cc8496-h2pnq"]
Sep 17 17:40:20.371800 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.371782    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-769cc8496-h2pnq"]
Sep 17 17:40:20.897416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:40:20.897384    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b196eac0-fc37-401d-8d07-70411e4e3d1f" path="/var/lib/kubelet/pods/b196eac0-fc37-401d-8d07-70411e4e3d1f/volumes"
Sep 17 17:41:25.216397 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.216361    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"]
Sep 17 17:41:25.216828 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.216669    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b196eac0-fc37-401d-8d07-70411e4e3d1f" containerName="console"
Sep 17 17:41:25.216828 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.216680    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b196eac0-fc37-401d-8d07-70411e4e3d1f" containerName="console"
Sep 17 17:41:25.216828 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.216758    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b196eac0-fc37-401d-8d07-70411e4e3d1f" containerName="console"
Sep 17 17:41:25.219934 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.219916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.222620 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.222596    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dwls2\""
Sep 17 17:41:25.222746 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.222599    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:41:25.223634 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.223614    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:41:25.228344 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.228321    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"]
Sep 17 17:41:25.270467 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.270441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.270607 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.270485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.270607 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.270509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8s2v\" (UniqueName: \"kubernetes.io/projected/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-kube-api-access-h8s2v\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.371188 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.371139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.371345 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.371216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.371345 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.371248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8s2v\" (UniqueName: \"kubernetes.io/projected/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-kube-api-access-h8s2v\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.371525 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.371507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.371593 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.371575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.381394 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.381365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8s2v\" (UniqueName: \"kubernetes.io/projected/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-kube-api-access-h8s2v\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.529338 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.529312    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:25.802890 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.802867    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"]
Sep 17 17:41:25.805392 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:41:25.805368    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c52e02_18d6_4f2d_b615_a6312ddf8af7.slice/crio-c99b8a221dd2ed3a85e3f3750db3024a3cf50fe2874ceed90176811e733fac5c WatchSource:0}: Error finding container c99b8a221dd2ed3a85e3f3750db3024a3cf50fe2874ceed90176811e733fac5c: Status 404 returned error can't find the container with id c99b8a221dd2ed3a85e3f3750db3024a3cf50fe2874ceed90176811e733fac5c
Sep 17 17:41:25.807377 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:25.807356    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:26.600125 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:26.600086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz" event={"ID":"b4c52e02-18d6-4f2d-b615-a6312ddf8af7","Type":"ContainerStarted","Data":"c99b8a221dd2ed3a85e3f3750db3024a3cf50fe2874ceed90176811e733fac5c"}
Sep 17 17:41:32.628797 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:32.628763    2581 generic.go:356] "Generic (PLEG): container finished" podID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerID="9b4c859b7e3ed995e7306815242fe6fa760e01c5ff9aed60365cb5f850300381" exitCode=0
Sep 17 17:41:32.629218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:32.628851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz" event={"ID":"b4c52e02-18d6-4f2d-b615-a6312ddf8af7","Type":"ContainerDied","Data":"9b4c859b7e3ed995e7306815242fe6fa760e01c5ff9aed60365cb5f850300381"}
Sep 17 17:41:41.668381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:41.668287    2581 generic.go:356] "Generic (PLEG): container finished" podID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerID="1876cfe51ad0d077b758f610074947db443a7735ee7ee8eddd275e4a3168ecea" exitCode=0
Sep 17 17:41:41.668381 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:41.668365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz" event={"ID":"b4c52e02-18d6-4f2d-b615-a6312ddf8af7","Type":"ContainerDied","Data":"1876cfe51ad0d077b758f610074947db443a7735ee7ee8eddd275e4a3168ecea"}
Sep 17 17:41:48.709139 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:48.709098    2581 generic.go:356] "Generic (PLEG): container finished" podID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerID="c2f96b00ea911027be864bd488a057b5a24c6e1fb0a67badafe8f52d48c3e484" exitCode=0
Sep 17 17:41:48.709139 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:48.709139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz" event={"ID":"b4c52e02-18d6-4f2d-b615-a6312ddf8af7","Type":"ContainerDied","Data":"c2f96b00ea911027be864bd488a057b5a24c6e1fb0a67badafe8f52d48c3e484"}
Sep 17 17:41:49.832171 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.832145    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:49.884152 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.884122    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-util\") pod \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") "
Sep 17 17:41:49.884272 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.884171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-bundle\") pod \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") "
Sep 17 17:41:49.884272 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.884214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8s2v\" (UniqueName: \"kubernetes.io/projected/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-kube-api-access-h8s2v\") pod \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\" (UID: \"b4c52e02-18d6-4f2d-b615-a6312ddf8af7\") "
Sep 17 17:41:49.884803 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.884778    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-bundle" (OuterVolumeSpecName: "bundle") pod "b4c52e02-18d6-4f2d-b615-a6312ddf8af7" (UID: "b4c52e02-18d6-4f2d-b615-a6312ddf8af7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:41:49.886393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.886366    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-kube-api-access-h8s2v" (OuterVolumeSpecName: "kube-api-access-h8s2v") pod "b4c52e02-18d6-4f2d-b615-a6312ddf8af7" (UID: "b4c52e02-18d6-4f2d-b615-a6312ddf8af7"). InnerVolumeSpecName "kube-api-access-h8s2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:41:49.888974 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.888953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-util" (OuterVolumeSpecName: "util") pod "b4c52e02-18d6-4f2d-b615-a6312ddf8af7" (UID: "b4c52e02-18d6-4f2d-b615-a6312ddf8af7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:41:49.985811 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.985743    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8s2v\" (UniqueName: \"kubernetes.io/projected/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-kube-api-access-h8s2v\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:41:49.985811 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.985765    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-util\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:41:49.985811 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:49.985776    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c52e02-18d6-4f2d-b615-a6312ddf8af7-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:41:50.719420 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:50.719389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz" event={"ID":"b4c52e02-18d6-4f2d-b615-a6312ddf8af7","Type":"ContainerDied","Data":"c99b8a221dd2ed3a85e3f3750db3024a3cf50fe2874ceed90176811e733fac5c"}
Sep 17 17:41:50.719420 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:50.719422    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1agk8mz"
Sep 17 17:41:50.719630 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:50.719432    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c99b8a221dd2ed3a85e3f3750db3024a3cf50fe2874ceed90176811e733fac5c"
Sep 17 17:41:58.936160 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.936082    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jvg4b"]
Sep 17 17:41:58.936512 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.936377    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerName="pull"
Sep 17 17:41:58.936512 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.936388    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerName="pull"
Sep 17 17:41:58.936512 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.936402    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerName="extract"
Sep 17 17:41:58.936512 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.936408    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerName="extract"
Sep 17 17:41:58.936512 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.936421    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerName="util"
Sep 17 17:41:58.936512 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.936426    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerName="util"
Sep 17 17:41:58.936512 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.936472    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c52e02-18d6-4f2d-b615-a6312ddf8af7" containerName="extract"
Sep 17 17:41:58.938313 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.938296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:41:58.941177 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.941157    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:41:58.941290 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.941154    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 17:41:58.942548 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.942533    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:41:58.942642 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.942566    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:41:58.942642 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.942569    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-g6smt\""
Sep 17 17:41:58.951395 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:58.951367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jvg4b"]
Sep 17 17:41:59.061170 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.061134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjpsn\" (UniqueName: \"kubernetes.io/projected/2e47a372-6ec7-41e7-826d-02a107acb947-kube-api-access-mjpsn\") pod \"keda-admission-6c4d864b86-jvg4b\" (UID: \"2e47a372-6ec7-41e7-826d-02a107acb947\") " pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:41:59.061535 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.061515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2e47a372-6ec7-41e7-826d-02a107acb947-certificates\") pod \"keda-admission-6c4d864b86-jvg4b\" (UID: \"2e47a372-6ec7-41e7-826d-02a107acb947\") " pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:41:59.162593 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.162553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2e47a372-6ec7-41e7-826d-02a107acb947-certificates\") pod \"keda-admission-6c4d864b86-jvg4b\" (UID: \"2e47a372-6ec7-41e7-826d-02a107acb947\") " pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:41:59.162775 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.162637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjpsn\" (UniqueName: \"kubernetes.io/projected/2e47a372-6ec7-41e7-826d-02a107acb947-kube-api-access-mjpsn\") pod \"keda-admission-6c4d864b86-jvg4b\" (UID: \"2e47a372-6ec7-41e7-826d-02a107acb947\") " pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:41:59.165633 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.165582    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2e47a372-6ec7-41e7-826d-02a107acb947-certificates\") pod \"keda-admission-6c4d864b86-jvg4b\" (UID: \"2e47a372-6ec7-41e7-826d-02a107acb947\") " pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:41:59.174876 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.174850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjpsn\" (UniqueName: \"kubernetes.io/projected/2e47a372-6ec7-41e7-826d-02a107acb947-kube-api-access-mjpsn\") pod \"keda-admission-6c4d864b86-jvg4b\" (UID: \"2e47a372-6ec7-41e7-826d-02a107acb947\") " pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:41:59.248458 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.248427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:41:59.529100 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.529010    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jvg4b"]
Sep 17 17:41:59.532068 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:41:59.532034    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e47a372_6ec7_41e7_826d_02a107acb947.slice/crio-208f699b2c57b8f1a4d666d40134f3ac63f4ed93d7e6c4d34035147df9930e38 WatchSource:0}: Error finding container 208f699b2c57b8f1a4d666d40134f3ac63f4ed93d7e6c4d34035147df9930e38: Status 404 returned error can't find the container with id 208f699b2c57b8f1a4d666d40134f3ac63f4ed93d7e6c4d34035147df9930e38
Sep 17 17:41:59.757103 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:41:59.757067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jvg4b" event={"ID":"2e47a372-6ec7-41e7-826d-02a107acb947","Type":"ContainerStarted","Data":"208f699b2c57b8f1a4d666d40134f3ac63f4ed93d7e6c4d34035147df9930e38"}
Sep 17 17:42:01.766349 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:01.766313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jvg4b" event={"ID":"2e47a372-6ec7-41e7-826d-02a107acb947","Type":"ContainerStarted","Data":"39d0f151fc0970fcff1fd13862660a683cc5fd75fc97588954dc7b7957a5e344"}
Sep 17 17:42:01.766809 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:01.766418    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:42:01.789685 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:01.789632    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-jvg4b" podStartSLOduration=2.052278017 podStartE2EDuration="3.789616297s" totalImagesPullingTime="1.73733828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:01.787412961 +0000 UTC m=+437.019443780" watchObservedRunningTime="2026-09-17 17:42:01.789616297 +0000 UTC m=+437.021647116"
Sep 17 17:42:22.773171 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:22.773140    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-jvg4b"
Sep 17 17:42:28.087746 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.087697    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dc75565b7-v4kdl"]
Sep 17 17:42:28.093321 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.090987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.104206 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.104175    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dc75565b7-v4kdl"]
Sep 17 17:42:28.180662 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.180625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-service-ca\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.180840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.180697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-config\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.180840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.180787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-oauth-config\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.180840 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.180824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfzj\" (UniqueName: \"kubernetes.io/projected/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-kube-api-access-lqfzj\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.181012 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.180856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-trusted-ca-bundle\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.181012 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.180912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-oauth-serving-cert\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.181012 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.180946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-serving-cert\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.282105 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.282074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-serving-cert\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.282284 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.282119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-service-ca\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.282284 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.282161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-config\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.282284 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.282183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-oauth-config\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.282284 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.282201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfzj\" (UniqueName: \"kubernetes.io/projected/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-kube-api-access-lqfzj\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.282284 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.282221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-trusted-ca-bundle\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.282284 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.282255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-oauth-serving-cert\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.282967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.282940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-service-ca\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.283070 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.283051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-oauth-serving-cert\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.283070 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.283059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-config\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.283169 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.283151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-trusted-ca-bundle\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.285131 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.285102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-oauth-config\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.285339 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.285319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-console-serving-cert\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.291528 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.291504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfzj\" (UniqueName: \"kubernetes.io/projected/ef1a9e76-99c3-4599-a28c-96f45fd0eac1-kube-api-access-lqfzj\") pod \"console-5dc75565b7-v4kdl\" (UID: \"ef1a9e76-99c3-4599-a28c-96f45fd0eac1\") " pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.403664 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.403594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:28.678650 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.678622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dc75565b7-v4kdl"]
Sep 17 17:42:28.680599 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:42:28.680572    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1a9e76_99c3_4599_a28c_96f45fd0eac1.slice/crio-8258ad71006f88b4a6b3d5019b7fb1329cc3e81839353e9d52677340fad7e9b4 WatchSource:0}: Error finding container 8258ad71006f88b4a6b3d5019b7fb1329cc3e81839353e9d52677340fad7e9b4: Status 404 returned error can't find the container with id 8258ad71006f88b4a6b3d5019b7fb1329cc3e81839353e9d52677340fad7e9b4
Sep 17 17:42:28.877721 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.877679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc75565b7-v4kdl" event={"ID":"ef1a9e76-99c3-4599-a28c-96f45fd0eac1","Type":"ContainerStarted","Data":"7add57f1e0e2feae57d1f8ba2ab30c70742b0e4a7567882318f8b09090f613ac"}
Sep 17 17:42:28.877906 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.877740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc75565b7-v4kdl" event={"ID":"ef1a9e76-99c3-4599-a28c-96f45fd0eac1","Type":"ContainerStarted","Data":"8258ad71006f88b4a6b3d5019b7fb1329cc3e81839353e9d52677340fad7e9b4"}
Sep 17 17:42:28.901024 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:28.900975    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dc75565b7-v4kdl" podStartSLOduration=0.900959997 podStartE2EDuration="900.959997ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:28.898514563 +0000 UTC m=+464.130545382" watchObservedRunningTime="2026-09-17 17:42:28.900959997 +0000 UTC m=+464.132990816"
Sep 17 17:42:38.403801 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:38.403769    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:38.404192 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:38.403820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:38.408765 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:38.408746    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:38.922429 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:38.922401    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dc75565b7-v4kdl"
Sep 17 17:42:38.983266 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:42:38.983232    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b98ffbbf9-nj6ht"]
Sep 17 17:43:04.004722 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.004649    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b98ffbbf9-nj6ht" podUID="41f0a15b-1f10-4d44-a64d-4a34a126c074" containerName="console" containerID="cri-o://ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f" gracePeriod=15
Sep 17 17:43:04.243196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.243166    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:43:04.381652 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.381581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-config\") pod \"41f0a15b-1f10-4d44-a64d-4a34a126c074\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") "
Sep 17 17:43:04.381652 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.381613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-service-ca\") pod \"41f0a15b-1f10-4d44-a64d-4a34a126c074\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") "
Sep 17 17:43:04.381652 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.381633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-oauth-serving-cert\") pod \"41f0a15b-1f10-4d44-a64d-4a34a126c074\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") "
Sep 17 17:43:04.381905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.381663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmzj7\" (UniqueName: \"kubernetes.io/projected/41f0a15b-1f10-4d44-a64d-4a34a126c074-kube-api-access-kmzj7\") pod \"41f0a15b-1f10-4d44-a64d-4a34a126c074\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") "
Sep 17 17:43:04.381905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.381762    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-oauth-config\") pod \"41f0a15b-1f10-4d44-a64d-4a34a126c074\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") "
Sep 17 17:43:04.381905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.381852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-serving-cert\") pod \"41f0a15b-1f10-4d44-a64d-4a34a126c074\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") "
Sep 17 17:43:04.381905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.381892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-trusted-ca-bundle\") pod \"41f0a15b-1f10-4d44-a64d-4a34a126c074\" (UID: \"41f0a15b-1f10-4d44-a64d-4a34a126c074\") "
Sep 17 17:43:04.382106 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.382011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-config" (OuterVolumeSpecName: "console-config") pod "41f0a15b-1f10-4d44-a64d-4a34a126c074" (UID: "41f0a15b-1f10-4d44-a64d-4a34a126c074"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:04.382106 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.382042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-service-ca" (OuterVolumeSpecName: "service-ca") pod "41f0a15b-1f10-4d44-a64d-4a34a126c074" (UID: "41f0a15b-1f10-4d44-a64d-4a34a126c074"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:04.382106 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.382076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "41f0a15b-1f10-4d44-a64d-4a34a126c074" (UID: "41f0a15b-1f10-4d44-a64d-4a34a126c074"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:04.382273 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.382255    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:43:04.382343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.382276    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-service-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:43:04.382343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.382289    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-oauth-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:43:04.382432 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.382365    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "41f0a15b-1f10-4d44-a64d-4a34a126c074" (UID: "41f0a15b-1f10-4d44-a64d-4a34a126c074"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:04.383975 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.383943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "41f0a15b-1f10-4d44-a64d-4a34a126c074" (UID: "41f0a15b-1f10-4d44-a64d-4a34a126c074"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:04.383975 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.383959    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "41f0a15b-1f10-4d44-a64d-4a34a126c074" (UID: "41f0a15b-1f10-4d44-a64d-4a34a126c074"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:04.384121 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.384020    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f0a15b-1f10-4d44-a64d-4a34a126c074-kube-api-access-kmzj7" (OuterVolumeSpecName: "kube-api-access-kmzj7") pod "41f0a15b-1f10-4d44-a64d-4a34a126c074" (UID: "41f0a15b-1f10-4d44-a64d-4a34a126c074"). InnerVolumeSpecName "kube-api-access-kmzj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:43:04.483677 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.483639    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmzj7\" (UniqueName: \"kubernetes.io/projected/41f0a15b-1f10-4d44-a64d-4a34a126c074-kube-api-access-kmzj7\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:43:04.483677 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.483669    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-oauth-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:43:04.483677 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.483678    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41f0a15b-1f10-4d44-a64d-4a34a126c074-console-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:43:04.483677 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:04.483687    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f0a15b-1f10-4d44-a64d-4a34a126c074-trusted-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:43:05.022862 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.022824    2581 generic.go:356] "Generic (PLEG): container finished" podID="41f0a15b-1f10-4d44-a64d-4a34a126c074" containerID="ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f" exitCode=0
Sep 17 17:43:05.023266 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.022875    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b98ffbbf9-nj6ht"
Sep 17 17:43:05.023266 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.022904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b98ffbbf9-nj6ht" event={"ID":"41f0a15b-1f10-4d44-a64d-4a34a126c074","Type":"ContainerDied","Data":"ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f"}
Sep 17 17:43:05.023266 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.022970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b98ffbbf9-nj6ht" event={"ID":"41f0a15b-1f10-4d44-a64d-4a34a126c074","Type":"ContainerDied","Data":"01d326c0eeb563e91784fc4de07bccc168e4b5d4822d3c9d0da47cf83f84c58f"}
Sep 17 17:43:05.023266 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.022990    2581 scope.go:117] "RemoveContainer" containerID="ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f"
Sep 17 17:43:05.031309 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.031280    2581 scope.go:117] "RemoveContainer" containerID="ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f"
Sep 17 17:43:05.031553 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:43:05.031531    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f\": container with ID starting with ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f not found: ID does not exist" containerID="ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f"
Sep 17 17:43:05.031622 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.031564    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f"} err="failed to get container status \"ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f\": rpc error: code = NotFound desc = could not find container \"ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f\": container with ID starting with ab82b948ccd825d818de9bc38a8f01b713536e66b402d015f95f86822873ca7f not found: ID does not exist"
Sep 17 17:43:05.080895 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.080870    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b98ffbbf9-nj6ht"]
Sep 17 17:43:05.091603 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.091580    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b98ffbbf9-nj6ht"]
Sep 17 17:43:05.226239 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.226156    2581 patch_prober.go:28] interesting pod/console-6b98ffbbf9-nj6ht container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.27:8443/health\": context deadline exceeded" start-of-body=
Sep 17 17:43:05.226239 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:05.226225    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b98ffbbf9-nj6ht" podUID="41f0a15b-1f10-4d44-a64d-4a34a126c074" containerName="console" probeResult="failure" output="Get \"https://10.134.0.27:8443/health\": context deadline exceeded"
Sep 17 17:43:06.898110 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:06.898072    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f0a15b-1f10-4d44-a64d-4a34a126c074" path="/var/lib/kubelet/pods/41f0a15b-1f10-4d44-a64d-4a34a126c074/volumes"
Sep 17 17:43:38.459037 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.458964    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ndkhj"]
Sep 17 17:43:38.459419 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.459317    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f0a15b-1f10-4d44-a64d-4a34a126c074" containerName="console"
Sep 17 17:43:38.459419 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.459331    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f0a15b-1f10-4d44-a64d-4a34a126c074" containerName="console"
Sep 17 17:43:38.459419 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.459393    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f0a15b-1f10-4d44-a64d-4a34a126c074" containerName="console"
Sep 17 17:43:38.461195 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.461178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:38.478437 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.478414    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:43:38.479521 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.479505    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:43:38.479521 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.479515    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-g8n5s\""
Sep 17 17:43:38.479622 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.479506    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:43:38.488219 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.488193    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ndkhj"]
Sep 17 17:43:38.514498 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.514476    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-69zb9"]
Sep 17 17:43:38.516583 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.516569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:38.521635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.521619    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rrjxg\""
Sep 17 17:43:38.525186 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.525170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:43:38.542095 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.542070    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-69zb9"]
Sep 17 17:43:38.554295 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.554276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09303678-d1e4-4d58-86fe-b04f19ecb3d3-cert\") pod \"kserve-controller-manager-7f654675b-ndkhj\" (UID: \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\") " pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:38.554393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.554306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwdx\" (UniqueName: \"kubernetes.io/projected/e6210316-d84d-490b-b92b-7efa0c61438c-kube-api-access-dpwdx\") pod \"seaweedfs-64568bcd49-69zb9\" (UID: \"e6210316-d84d-490b-b92b-7efa0c61438c\") " pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:38.554393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.554359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6210316-d84d-490b-b92b-7efa0c61438c-data\") pod \"seaweedfs-64568bcd49-69zb9\" (UID: \"e6210316-d84d-490b-b92b-7efa0c61438c\") " pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:38.554393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.554387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgz2q\" (UniqueName: \"kubernetes.io/projected/09303678-d1e4-4d58-86fe-b04f19ecb3d3-kube-api-access-fgz2q\") pod \"kserve-controller-manager-7f654675b-ndkhj\" (UID: \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\") " pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:38.655256 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.655218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwdx\" (UniqueName: \"kubernetes.io/projected/e6210316-d84d-490b-b92b-7efa0c61438c-kube-api-access-dpwdx\") pod \"seaweedfs-64568bcd49-69zb9\" (UID: \"e6210316-d84d-490b-b92b-7efa0c61438c\") " pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:38.655444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.655308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6210316-d84d-490b-b92b-7efa0c61438c-data\") pod \"seaweedfs-64568bcd49-69zb9\" (UID: \"e6210316-d84d-490b-b92b-7efa0c61438c\") " pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:38.655444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.655346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgz2q\" (UniqueName: \"kubernetes.io/projected/09303678-d1e4-4d58-86fe-b04f19ecb3d3-kube-api-access-fgz2q\") pod \"kserve-controller-manager-7f654675b-ndkhj\" (UID: \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\") " pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:38.655444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.655379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09303678-d1e4-4d58-86fe-b04f19ecb3d3-cert\") pod \"kserve-controller-manager-7f654675b-ndkhj\" (UID: \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\") " pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:38.655753 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.655730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6210316-d84d-490b-b92b-7efa0c61438c-data\") pod \"seaweedfs-64568bcd49-69zb9\" (UID: \"e6210316-d84d-490b-b92b-7efa0c61438c\") " pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:38.657791 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.657770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09303678-d1e4-4d58-86fe-b04f19ecb3d3-cert\") pod \"kserve-controller-manager-7f654675b-ndkhj\" (UID: \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\") " pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:38.672947 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.672912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgz2q\" (UniqueName: \"kubernetes.io/projected/09303678-d1e4-4d58-86fe-b04f19ecb3d3-kube-api-access-fgz2q\") pod \"kserve-controller-manager-7f654675b-ndkhj\" (UID: \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\") " pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:38.674645 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.674627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwdx\" (UniqueName: \"kubernetes.io/projected/e6210316-d84d-490b-b92b-7efa0c61438c-kube-api-access-dpwdx\") pod \"seaweedfs-64568bcd49-69zb9\" (UID: \"e6210316-d84d-490b-b92b-7efa0c61438c\") " pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:38.770562 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.770534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:38.825584 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:38.825552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:39.045357 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:39.045330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ndkhj"]
Sep 17 17:43:39.047157 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:43:39.047131    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09303678_d1e4_4d58_86fe_b04f19ecb3d3.slice/crio-4d00c1b4220e52a1418c17c26068423be42b4ae3292b01bb3b8d594b30518e69 WatchSource:0}: Error finding container 4d00c1b4220e52a1418c17c26068423be42b4ae3292b01bb3b8d594b30518e69: Status 404 returned error can't find the container with id 4d00c1b4220e52a1418c17c26068423be42b4ae3292b01bb3b8d594b30518e69
Sep 17 17:43:39.107831 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:39.107805    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-69zb9"]
Sep 17 17:43:39.109623 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:43:39.109600    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6210316_d84d_490b_b92b_7efa0c61438c.slice/crio-6bbfe867bb5011f5b6ba5a1edeb721ff37bd4c1a055ffa901ca0d6d53eca84f4 WatchSource:0}: Error finding container 6bbfe867bb5011f5b6ba5a1edeb721ff37bd4c1a055ffa901ca0d6d53eca84f4: Status 404 returned error can't find the container with id 6bbfe867bb5011f5b6ba5a1edeb721ff37bd4c1a055ffa901ca0d6d53eca84f4
Sep 17 17:43:39.167967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:39.167934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-69zb9" event={"ID":"e6210316-d84d-490b-b92b-7efa0c61438c","Type":"ContainerStarted","Data":"6bbfe867bb5011f5b6ba5a1edeb721ff37bd4c1a055ffa901ca0d6d53eca84f4"}
Sep 17 17:43:39.169084 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:39.169061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-ndkhj" event={"ID":"09303678-d1e4-4d58-86fe-b04f19ecb3d3","Type":"ContainerStarted","Data":"4d00c1b4220e52a1418c17c26068423be42b4ae3292b01bb3b8d594b30518e69"}
Sep 17 17:43:43.194168 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:43.194133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-69zb9" event={"ID":"e6210316-d84d-490b-b92b-7efa0c61438c","Type":"ContainerStarted","Data":"bfb8343876c1cf5591353aab2d783696f276db124db634edb45ae3d0b97e8276"}
Sep 17 17:43:43.194638 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:43.194225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:43:43.196159 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:43.196136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-ndkhj" event={"ID":"09303678-d1e4-4d58-86fe-b04f19ecb3d3","Type":"ContainerStarted","Data":"451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5"}
Sep 17 17:43:43.196345 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:43.196332    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:43:43.263690 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:43.263642    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-69zb9" podStartSLOduration=1.725001066 podStartE2EDuration="5.263624838s" totalImagesPullingTime="3.538623772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:43.261438417 +0000 UTC m=+538.493469236" watchObservedRunningTime="2026-09-17 17:43:43.263624838 +0000 UTC m=+538.495655657"
Sep 17 17:43:43.311649 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:43.311607    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f654675b-ndkhj" podStartSLOduration=1.796340332 podStartE2EDuration="5.311591822s" totalImagesPullingTime="3.51525149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:43.310888755 +0000 UTC m=+538.542919589" watchObservedRunningTime="2026-09-17 17:43:43.311591822 +0000 UTC m=+538.543622640"
Sep 17 17:43:49.204018 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:43:49.203989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-69zb9"
Sep 17 17:44:14.206668 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:14.206638    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:44:23.590586 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.590553    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ndkhj"]
Sep 17 17:44:23.591006 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.590758    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7f654675b-ndkhj" podUID="09303678-d1e4-4d58-86fe-b04f19ecb3d3" containerName="manager" containerID="cri-o://451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5" gracePeriod=10
Sep 17 17:44:23.623791 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.623764    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f654675b-cc47q"]
Sep 17 17:44:23.627393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.627372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:23.635722 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.635678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-cc47q"]
Sep 17 17:44:23.727754 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.727725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxqx4\" (UniqueName: \"kubernetes.io/projected/f33aad09-e3ee-4bfb-b420-9f6765e40139-kube-api-access-fxqx4\") pod \"kserve-controller-manager-7f654675b-cc47q\" (UID: \"f33aad09-e3ee-4bfb-b420-9f6765e40139\") " pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:23.727865 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.727760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33aad09-e3ee-4bfb-b420-9f6765e40139-cert\") pod \"kserve-controller-manager-7f654675b-cc47q\" (UID: \"f33aad09-e3ee-4bfb-b420-9f6765e40139\") " pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:23.820828 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.820805    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:44:23.828380 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.828354    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgz2q\" (UniqueName: \"kubernetes.io/projected/09303678-d1e4-4d58-86fe-b04f19ecb3d3-kube-api-access-fgz2q\") pod \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\" (UID: \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\") "
Sep 17 17:44:23.828489 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.828466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09303678-d1e4-4d58-86fe-b04f19ecb3d3-cert\") pod \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\" (UID: \"09303678-d1e4-4d58-86fe-b04f19ecb3d3\") "
Sep 17 17:44:23.828654 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.828636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxqx4\" (UniqueName: \"kubernetes.io/projected/f33aad09-e3ee-4bfb-b420-9f6765e40139-kube-api-access-fxqx4\") pod \"kserve-controller-manager-7f654675b-cc47q\" (UID: \"f33aad09-e3ee-4bfb-b420-9f6765e40139\") " pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:23.828782 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.828675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33aad09-e3ee-4bfb-b420-9f6765e40139-cert\") pod \"kserve-controller-manager-7f654675b-cc47q\" (UID: \"f33aad09-e3ee-4bfb-b420-9f6765e40139\") " pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:23.830556 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.830530    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09303678-d1e4-4d58-86fe-b04f19ecb3d3-kube-api-access-fgz2q" (OuterVolumeSpecName: "kube-api-access-fgz2q") pod "09303678-d1e4-4d58-86fe-b04f19ecb3d3" (UID: "09303678-d1e4-4d58-86fe-b04f19ecb3d3"). InnerVolumeSpecName "kube-api-access-fgz2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:23.830685 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.830665    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09303678-d1e4-4d58-86fe-b04f19ecb3d3-cert" (OuterVolumeSpecName: "cert") pod "09303678-d1e4-4d58-86fe-b04f19ecb3d3" (UID: "09303678-d1e4-4d58-86fe-b04f19ecb3d3"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:23.831142 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.831123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f33aad09-e3ee-4bfb-b420-9f6765e40139-cert\") pod \"kserve-controller-manager-7f654675b-cc47q\" (UID: \"f33aad09-e3ee-4bfb-b420-9f6765e40139\") " pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:23.838812 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.838793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxqx4\" (UniqueName: \"kubernetes.io/projected/f33aad09-e3ee-4bfb-b420-9f6765e40139-kube-api-access-fxqx4\") pod \"kserve-controller-manager-7f654675b-cc47q\" (UID: \"f33aad09-e3ee-4bfb-b420-9f6765e40139\") " pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:23.929975 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.929889    2581 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09303678-d1e4-4d58-86fe-b04f19ecb3d3-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:44:23.929975 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.929928    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgz2q\" (UniqueName: \"kubernetes.io/projected/09303678-d1e4-4d58-86fe-b04f19ecb3d3-kube-api-access-fgz2q\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:44:23.976836 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:23.976811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:24.252693 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.252669    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-cc47q"]
Sep 17 17:44:24.254784 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:44:24.254758    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33aad09_e3ee_4bfb_b420_9f6765e40139.slice/crio-46782d90dad0ccccbb6e66bcce48d5f41d330652cd8b698287010ce5f5df2e5a WatchSource:0}: Error finding container 46782d90dad0ccccbb6e66bcce48d5f41d330652cd8b698287010ce5f5df2e5a: Status 404 returned error can't find the container with id 46782d90dad0ccccbb6e66bcce48d5f41d330652cd8b698287010ce5f5df2e5a
Sep 17 17:44:24.372686 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.372652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-cc47q" event={"ID":"f33aad09-e3ee-4bfb-b420-9f6765e40139","Type":"ContainerStarted","Data":"46782d90dad0ccccbb6e66bcce48d5f41d330652cd8b698287010ce5f5df2e5a"}
Sep 17 17:44:24.373851 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.373824    2581 generic.go:356] "Generic (PLEG): container finished" podID="09303678-d1e4-4d58-86fe-b04f19ecb3d3" containerID="451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5" exitCode=0
Sep 17 17:44:24.373944 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.373871    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-ndkhj"
Sep 17 17:44:24.373944 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.373873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-ndkhj" event={"ID":"09303678-d1e4-4d58-86fe-b04f19ecb3d3","Type":"ContainerDied","Data":"451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5"}
Sep 17 17:44:24.374019 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.373975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-ndkhj" event={"ID":"09303678-d1e4-4d58-86fe-b04f19ecb3d3","Type":"ContainerDied","Data":"4d00c1b4220e52a1418c17c26068423be42b4ae3292b01bb3b8d594b30518e69"}
Sep 17 17:44:24.374019 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.374003    2581 scope.go:117] "RemoveContainer" containerID="451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5"
Sep 17 17:44:24.382238 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.382220    2581 scope.go:117] "RemoveContainer" containerID="451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5"
Sep 17 17:44:24.382521 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:44:24.382503    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5\": container with ID starting with 451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5 not found: ID does not exist" containerID="451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5"
Sep 17 17:44:24.382564 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.382531    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5"} err="failed to get container status \"451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5\": rpc error: code = NotFound desc = could not find container \"451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5\": container with ID starting with 451466ef2399c74aad2df47e2b5ba96aedfa75fe19d240105fbed3dd8f74d9e5 not found: ID does not exist"
Sep 17 17:44:24.406678 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.406653    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ndkhj"]
Sep 17 17:44:24.411653 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.411633    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ndkhj"]
Sep 17 17:44:24.897837 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:24.897816    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09303678-d1e4-4d58-86fe-b04f19ecb3d3" path="/var/lib/kubelet/pods/09303678-d1e4-4d58-86fe-b04f19ecb3d3/volumes"
Sep 17 17:44:25.380949 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:25.380907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-cc47q" event={"ID":"f33aad09-e3ee-4bfb-b420-9f6765e40139","Type":"ContainerStarted","Data":"88468cb735e4f1b8439ec12539fcf0f9fd83284d63a4bd8f3ef9004f2fd1cd70"}
Sep 17 17:44:25.381129 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:25.381035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:25.405810 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:25.405757    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f654675b-cc47q" podStartSLOduration=1.805564941 podStartE2EDuration="2.40574171s" totalImagesPullingTime="600.176769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:25.404046347 +0000 UTC m=+580.636077165" watchObservedRunningTime="2026-09-17 17:44:25.40574171 +0000 UTC m=+580.637772580"
Sep 17 17:44:46.777874 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:46.777843    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:44:46.782183 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:46.782161    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:44:56.390180 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:56.390153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f654675b-cc47q"
Sep 17 17:44:57.895074 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.894995    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-wm4k8"]
Sep 17 17:44:57.895493 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.895477    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09303678-d1e4-4d58-86fe-b04f19ecb3d3" containerName="manager"
Sep 17 17:44:57.895546 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.895496    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09303678-d1e4-4d58-86fe-b04f19ecb3d3" containerName="manager"
Sep 17 17:44:57.895584 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.895549    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="09303678-d1e4-4d58-86fe-b04f19ecb3d3" containerName="manager"
Sep 17 17:44:57.898275 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.898260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:44:57.901681 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.901656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 17:44:57.901927 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.901912    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-c92d6\""
Sep 17 17:44:57.922396 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.922371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wm4k8"]
Sep 17 17:44:57.930439 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.930420    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-vlvp7"]
Sep 17 17:44:57.933532 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.933517    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:57.937599 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.937582    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 17:44:57.937732 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.937699    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5v64s\""
Sep 17 17:44:57.938521 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.938505    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 17:44:57.945985 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.945966    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vlvp7"]
Sep 17 17:44:57.996395 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.996376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/257b18e4-d825-4749-a2d5-229d00cf61c4-cert\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:57.996493 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.996425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e860015c-8390-41bd-a4a1-e191c1bc3faa-tls-certs\") pod \"model-serving-api-7b954ffc95-wm4k8\" (UID: \"e860015c-8390-41bd-a4a1-e191c1bc3faa\") " pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:44:57.996493 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.996452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/257b18e4-d825-4749-a2d5-229d00cf61c4-metrics-certs\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:57.996567 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.996514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxqp\" (UniqueName: \"kubernetes.io/projected/257b18e4-d825-4749-a2d5-229d00cf61c4-kube-api-access-mlxqp\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:57.996567 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:57.996540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsn4p\" (UniqueName: \"kubernetes.io/projected/e860015c-8390-41bd-a4a1-e191c1bc3faa-kube-api-access-wsn4p\") pod \"model-serving-api-7b954ffc95-wm4k8\" (UID: \"e860015c-8390-41bd-a4a1-e191c1bc3faa\") " pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:44:58.097561 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.097536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e860015c-8390-41bd-a4a1-e191c1bc3faa-tls-certs\") pod \"model-serving-api-7b954ffc95-wm4k8\" (UID: \"e860015c-8390-41bd-a4a1-e191c1bc3faa\") " pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:44:58.097657 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.097573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/257b18e4-d825-4749-a2d5-229d00cf61c4-metrics-certs\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:58.097657 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.097607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxqp\" (UniqueName: \"kubernetes.io/projected/257b18e4-d825-4749-a2d5-229d00cf61c4-kube-api-access-mlxqp\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:58.097657 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.097634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsn4p\" (UniqueName: \"kubernetes.io/projected/e860015c-8390-41bd-a4a1-e191c1bc3faa-kube-api-access-wsn4p\") pod \"model-serving-api-7b954ffc95-wm4k8\" (UID: \"e860015c-8390-41bd-a4a1-e191c1bc3faa\") " pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:44:58.097807 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.097675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/257b18e4-d825-4749-a2d5-229d00cf61c4-cert\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:58.100114 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.100094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/257b18e4-d825-4749-a2d5-229d00cf61c4-metrics-certs\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:58.100259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.100238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e860015c-8390-41bd-a4a1-e191c1bc3faa-tls-certs\") pod \"model-serving-api-7b954ffc95-wm4k8\" (UID: \"e860015c-8390-41bd-a4a1-e191c1bc3faa\") " pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:44:58.100319 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.100238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/257b18e4-d825-4749-a2d5-229d00cf61c4-cert\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:58.111847 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.111824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxqp\" (UniqueName: \"kubernetes.io/projected/257b18e4-d825-4749-a2d5-229d00cf61c4-kube-api-access-mlxqp\") pod \"odh-model-controller-7b966d7984-vlvp7\" (UID: \"257b18e4-d825-4749-a2d5-229d00cf61c4\") " pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:58.114570 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.114547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsn4p\" (UniqueName: \"kubernetes.io/projected/e860015c-8390-41bd-a4a1-e191c1bc3faa-kube-api-access-wsn4p\") pod \"model-serving-api-7b954ffc95-wm4k8\" (UID: \"e860015c-8390-41bd-a4a1-e191c1bc3faa\") " pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:44:58.207730 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.207665    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:44:58.242885 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.242863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:44:58.489689 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.489667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wm4k8"]
Sep 17 17:44:58.491913 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:44:58.491891    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode860015c_8390_41bd_a4a1_e191c1bc3faa.slice/crio-81489dcf136c4639fe4b85803e9f597f1afdd3f6371b0d9dfddadedc6d28e557 WatchSource:0}: Error finding container 81489dcf136c4639fe4b85803e9f597f1afdd3f6371b0d9dfddadedc6d28e557: Status 404 returned error can't find the container with id 81489dcf136c4639fe4b85803e9f597f1afdd3f6371b0d9dfddadedc6d28e557
Sep 17 17:44:58.521009 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.520987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wm4k8" event={"ID":"e860015c-8390-41bd-a4a1-e191c1bc3faa","Type":"ContainerStarted","Data":"81489dcf136c4639fe4b85803e9f597f1afdd3f6371b0d9dfddadedc6d28e557"}
Sep 17 17:44:58.530177 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:58.530156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vlvp7"]
Sep 17 17:44:58.532972 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:44:58.532950    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257b18e4_d825_4749_a2d5_229d00cf61c4.slice/crio-9aedce997d2c369a47b8b7b55ee6d73921c1ac4cce0f02d75153cf34abf2b972 WatchSource:0}: Error finding container 9aedce997d2c369a47b8b7b55ee6d73921c1ac4cce0f02d75153cf34abf2b972: Status 404 returned error can't find the container with id 9aedce997d2c369a47b8b7b55ee6d73921c1ac4cce0f02d75153cf34abf2b972
Sep 17 17:44:59.535629 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:44:59.535554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vlvp7" event={"ID":"257b18e4-d825-4749-a2d5-229d00cf61c4","Type":"ContainerStarted","Data":"9aedce997d2c369a47b8b7b55ee6d73921c1ac4cce0f02d75153cf34abf2b972"}
Sep 17 17:45:02.551857 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:45:02.551825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vlvp7" event={"ID":"257b18e4-d825-4749-a2d5-229d00cf61c4","Type":"ContainerStarted","Data":"c82e40966512e859bc7a9d3d5a742ede83240cbe52122824b1c31dc17408eebe"}
Sep 17 17:45:02.552400 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:45:02.551915    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:45:02.553462 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:45:02.553436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wm4k8" event={"ID":"e860015c-8390-41bd-a4a1-e191c1bc3faa","Type":"ContainerStarted","Data":"1fe32f17259b8fca591ee8aea11e5e25e233b68e2fa84d7779ba5f657f3e1424"}
Sep 17 17:45:02.553586 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:45:02.553560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:45:02.645090 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:45:02.645043    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-vlvp7" podStartSLOduration=2.164690781 podStartE2EDuration="5.645027768s" totalImagesPullingTime="3.480336987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:02.644547642 +0000 UTC m=+617.876578463" watchObservedRunningTime="2026-09-17 17:45:02.645027768 +0000 UTC m=+617.877058630"
Sep 17 17:45:02.696676 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:45:02.696636    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-wm4k8" podStartSLOduration=2.222096522 podStartE2EDuration="5.696621742s" totalImagesPullingTime="3.47452522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:02.69440239 +0000 UTC m=+617.926433208" watchObservedRunningTime="2026-09-17 17:45:02.696621742 +0000 UTC m=+617.928652560"
Sep 17 17:45:13.561390 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:45:13.561357    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-vlvp7"
Sep 17 17:45:13.563151 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:45:13.563133    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-wm4k8"
Sep 17 17:46:45.464766 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.464682    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"]
Sep 17 17:46:45.468191 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.468173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:45.470952 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.470933    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4a129-predictor-serving-cert\""
Sep 17 17:46:45.471070 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.470974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:46:45.471762 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.471748    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:46:45.472198 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.472178    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4a129-kube-rbac-proxy-sar-config\""
Sep 17 17:46:45.472243 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.472182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-r96lh\""
Sep 17 17:46:45.487729 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.487687    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"]
Sep 17 17:46:45.593109 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.593072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx5rg\" (UniqueName: \"kubernetes.io/projected/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-api-access-fx5rg\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:45.593311 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.593112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:45.593311 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.593145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:45.694496 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.694464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx5rg\" (UniqueName: \"kubernetes.io/projected/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-api-access-fx5rg\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:45.694669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.694506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:45.694669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.694536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:45.694669 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:46:45.694624    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4a129-predictor-serving-cert: secret "error-404-isvc-4a129-predictor-serving-cert" not found
Sep 17 17:46:45.694855 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:46:45.694680    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls podName:209300bb-fba8-4b66-95fb-77b9d25418e4 nodeName:}" failed. No retries permitted until 2026-09-17 17:46:46.194665976 +0000 UTC m=+721.426696773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls") pod "error-404-isvc-4a129-predictor-579fd87984-zc225" (UID: "209300bb-fba8-4b66-95fb-77b9d25418e4") : secret "error-404-isvc-4a129-predictor-serving-cert" not found
Sep 17 17:46:45.695310 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.695288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:45.707677 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:45.707650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx5rg\" (UniqueName: \"kubernetes.io/projected/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-api-access-fx5rg\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:46.199852 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:46.199821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:46.202317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:46.202285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls\") pod \"error-404-isvc-4a129-predictor-579fd87984-zc225\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") " pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:46.379286 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:46.379250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:46:46.657785 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:46.657758    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"]
Sep 17 17:46:46.659333 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:46:46.659304    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209300bb_fba8_4b66_95fb_77b9d25418e4.slice/crio-e84150d444ae3879530836438b1ef7801bfb04184f6af01a5f023350dc26b4a9 WatchSource:0}: Error finding container e84150d444ae3879530836438b1ef7801bfb04184f6af01a5f023350dc26b4a9: Status 404 returned error can't find the container with id e84150d444ae3879530836438b1ef7801bfb04184f6af01a5f023350dc26b4a9
Sep 17 17:46:46.661337 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:46.661319    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:46:47.019714 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:47.019671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" event={"ID":"209300bb-fba8-4b66-95fb-77b9d25418e4","Type":"ContainerStarted","Data":"e84150d444ae3879530836438b1ef7801bfb04184f6af01a5f023350dc26b4a9"}
Sep 17 17:46:59.093754 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:46:59.093696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" event={"ID":"209300bb-fba8-4b66-95fb-77b9d25418e4","Type":"ContainerStarted","Data":"89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28"}
Sep 17 17:47:01.106483 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:01.106443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" event={"ID":"209300bb-fba8-4b66-95fb-77b9d25418e4","Type":"ContainerStarted","Data":"37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f"}
Sep 17 17:47:01.106917 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:01.106514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:47:01.106917 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:01.106569    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:47:01.107904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:01.107881    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:47:01.149914 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:01.149867    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podStartSLOduration=2.314911591 podStartE2EDuration="16.149853099s" totalImagesPullingTime="13.834941508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:01.144717984 +0000 UTC m=+736.376748793" watchObservedRunningTime="2026-09-17 17:47:01.149853099 +0000 UTC m=+736.381883918"
Sep 17 17:47:02.111330 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:02.111291    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:47:07.116642 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:07.116611    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:47:07.117090 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:07.117061    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:47:17.117464 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:17.117419    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:47:27.117670 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:27.117630    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:47:37.117072 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:37.117030    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:47:47.118566 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:47:47.118529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:48:04.997044 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:04.996973    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"]
Sep 17 17:48:04.999391 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:04.999375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:05.003057 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.003034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-4a129-serving-cert\""
Sep 17 17:48:05.003171 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.003038    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-4a129-kube-rbac-proxy-sar-config\""
Sep 17 17:48:05.017605 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.017582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"]
Sep 17 17:48:05.137219 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.137191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9fbc618-64f3-4112-9eb7-a54419edbb5e-openshift-service-ca-bundle\") pod \"switch-graph-4a129-8fdcd5874-qjxjz\" (UID: \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\") " pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:05.137354 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.137236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9fbc618-64f3-4112-9eb7-a54419edbb5e-proxy-tls\") pod \"switch-graph-4a129-8fdcd5874-qjxjz\" (UID: \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\") " pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:05.238510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.238481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9fbc618-64f3-4112-9eb7-a54419edbb5e-openshift-service-ca-bundle\") pod \"switch-graph-4a129-8fdcd5874-qjxjz\" (UID: \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\") " pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:05.238633 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.238518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9fbc618-64f3-4112-9eb7-a54419edbb5e-proxy-tls\") pod \"switch-graph-4a129-8fdcd5874-qjxjz\" (UID: \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\") " pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:05.239246 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.239224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9fbc618-64f3-4112-9eb7-a54419edbb5e-openshift-service-ca-bundle\") pod \"switch-graph-4a129-8fdcd5874-qjxjz\" (UID: \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\") " pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:05.240860 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.240845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9fbc618-64f3-4112-9eb7-a54419edbb5e-proxy-tls\") pod \"switch-graph-4a129-8fdcd5874-qjxjz\" (UID: \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\") " pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:05.309044 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.308986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:05.580905 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:05.580880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"]
Sep 17 17:48:05.582948 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:48:05.582921    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fbc618_64f3_4112_9eb7_a54419edbb5e.slice/crio-16a84e35561a02c8bbee3fb2f698347a68fbf2b3317560c1323724128e3bb711 WatchSource:0}: Error finding container 16a84e35561a02c8bbee3fb2f698347a68fbf2b3317560c1323724128e3bb711: Status 404 returned error can't find the container with id 16a84e35561a02c8bbee3fb2f698347a68fbf2b3317560c1323724128e3bb711
Sep 17 17:48:06.403189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:06.403146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" event={"ID":"b9fbc618-64f3-4112-9eb7-a54419edbb5e","Type":"ContainerStarted","Data":"16a84e35561a02c8bbee3fb2f698347a68fbf2b3317560c1323724128e3bb711"}
Sep 17 17:48:07.411873 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:07.411841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" event={"ID":"b9fbc618-64f3-4112-9eb7-a54419edbb5e","Type":"ContainerStarted","Data":"513c098f7407726f0260772ba5660587a848e6cfe9f26ea54aebfe2bd3147050"}
Sep 17 17:48:07.412197 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:07.411982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:07.445997 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:07.445957    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" podStartSLOduration=1.738621663 podStartE2EDuration="3.445943864s" totalImagesPullingTime="1.707322201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:07.44385915 +0000 UTC m=+802.675889970" watchObservedRunningTime="2026-09-17 17:48:07.445943864 +0000 UTC m=+802.677974727"
Sep 17 17:48:13.424213 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:13.424182    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:15.164734 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:15.164689    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"]
Sep 17 17:48:15.165137 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:15.164865    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129" containerID="cri-o://513c098f7407726f0260772ba5660587a848e6cfe9f26ea54aebfe2bd3147050" gracePeriod=30
Sep 17 17:48:15.444576 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:15.444498    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"]
Sep 17 17:48:15.444757 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:15.444732    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container" containerID="cri-o://89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28" gracePeriod=30
Sep 17 17:48:15.444802 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:15.444770    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kube-rbac-proxy" containerID="cri-o://37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f" gracePeriod=30
Sep 17 17:48:15.909054 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:15.909021    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"]
Sep 17 17:48:15.911765 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:15.911749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:15.914126 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:48:15.914093    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:15.914793 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:48:15.914765    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"error-404-isvc-8142b-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8142b-predictor-serving-cert\"" type="*v1.Secret"
Sep 17 17:48:15.914902 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:48:15.914880    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"error-404-isvc-8142b-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8142b-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 17 17:48:15.934555 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:15.934531    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"]
Sep 17 17:48:16.024521 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.024494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqkq\" (UniqueName: \"kubernetes.io/projected/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-api-access-wxqkq\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.024635 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.024555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.024681 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.024634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2409f72-3a11-4c01-b722-8b130754e5ae-proxy-tls\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.125509 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.125486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2409f72-3a11-4c01-b722-8b130754e5ae-proxy-tls\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.125617 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.125526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqkq\" (UniqueName: \"kubernetes.io/projected/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-api-access-wxqkq\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.125617 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.125573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.143109 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.143083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqkq\" (UniqueName: \"kubernetes.io/projected/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-api-access-wxqkq\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.457219 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.457182    2581 generic.go:356] "Generic (PLEG): container finished" podID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerID="37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f" exitCode=2
Sep 17 17:48:16.457567 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.457253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" event={"ID":"209300bb-fba8-4b66-95fb-77b9d25418e4","Type":"ContainerDied","Data":"37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f"}
Sep 17 17:48:16.751214 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.751188    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8142b-predictor-serving-cert\""
Sep 17 17:48:16.758509 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.758487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2409f72-3a11-4c01-b722-8b130754e5ae-proxy-tls\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.776218 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.776196    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8142b-kube-rbac-proxy-sar-config\""
Sep 17 17:48:16.786501 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.786479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") " pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:16.821819 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:16.821800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:17.107699 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.107634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"]
Sep 17 17:48:17.110993 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:48:17.110966    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2409f72_3a11_4c01_b722_8b130754e5ae.slice/crio-5751d1aa4259a577399909505b2a7319001143c24ea4f56b26e23e50a2b0390b WatchSource:0}: Error finding container 5751d1aa4259a577399909505b2a7319001143c24ea4f56b26e23e50a2b0390b: Status 404 returned error can't find the container with id 5751d1aa4259a577399909505b2a7319001143c24ea4f56b26e23e50a2b0390b
Sep 17 17:48:17.111895 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.111866    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 17:48:17.117363 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.117329    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:48:17.464234 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.464151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" event={"ID":"e2409f72-3a11-4c01-b722-8b130754e5ae","Type":"ContainerStarted","Data":"e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0"}
Sep 17 17:48:17.464234 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.464206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" event={"ID":"e2409f72-3a11-4c01-b722-8b130754e5ae","Type":"ContainerStarted","Data":"e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441"}
Sep 17 17:48:17.464234 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.464228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" event={"ID":"e2409f72-3a11-4c01-b722-8b130754e5ae","Type":"ContainerStarted","Data":"5751d1aa4259a577399909505b2a7319001143c24ea4f56b26e23e50a2b0390b"}
Sep 17 17:48:17.464619 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.464341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:17.464619 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.464375    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:17.465559 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.465536    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 17:48:17.504055 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:17.504001    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podStartSLOduration=2.503987038 podStartE2EDuration="2.503987038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:17.502798327 +0000 UTC m=+812.734829146" watchObservedRunningTime="2026-09-17 17:48:17.503987038 +0000 UTC m=+812.736017924"
Sep 17 17:48:18.179549 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.179527    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:48:18.346278 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.346199    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx5rg\" (UniqueName: \"kubernetes.io/projected/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-api-access-fx5rg\") pod \"209300bb-fba8-4b66-95fb-77b9d25418e4\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") "
Sep 17 17:48:18.346278 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.346263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-rbac-proxy-sar-config\") pod \"209300bb-fba8-4b66-95fb-77b9d25418e4\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") "
Sep 17 17:48:18.346461 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.346294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls\") pod \"209300bb-fba8-4b66-95fb-77b9d25418e4\" (UID: \"209300bb-fba8-4b66-95fb-77b9d25418e4\") "
Sep 17 17:48:18.346714 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.346678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "209300bb-fba8-4b66-95fb-77b9d25418e4" (UID: "209300bb-fba8-4b66-95fb-77b9d25418e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:48:18.348491 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.348463    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-api-access-fx5rg" (OuterVolumeSpecName: "kube-api-access-fx5rg") pod "209300bb-fba8-4b66-95fb-77b9d25418e4" (UID: "209300bb-fba8-4b66-95fb-77b9d25418e4"). InnerVolumeSpecName "kube-api-access-fx5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:18.348600 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.348515    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "209300bb-fba8-4b66-95fb-77b9d25418e4" (UID: "209300bb-fba8-4b66-95fb-77b9d25418e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:48:18.422151 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.422121    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:18.447514 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.447493    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx5rg\" (UniqueName: \"kubernetes.io/projected/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-api-access-fx5rg\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:48:18.447580 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.447513    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/209300bb-fba8-4b66-95fb-77b9d25418e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:48:18.447580 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.447524    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/209300bb-fba8-4b66-95fb-77b9d25418e4-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:48:18.470210 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.470183    2581 generic.go:356] "Generic (PLEG): container finished" podID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerID="89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28" exitCode=0
Sep 17 17:48:18.470590 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.470238    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"
Sep 17 17:48:18.470590 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.470269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" event={"ID":"209300bb-fba8-4b66-95fb-77b9d25418e4","Type":"ContainerDied","Data":"89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28"}
Sep 17 17:48:18.470590 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.470328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225" event={"ID":"209300bb-fba8-4b66-95fb-77b9d25418e4","Type":"ContainerDied","Data":"e84150d444ae3879530836438b1ef7801bfb04184f6af01a5f023350dc26b4a9"}
Sep 17 17:48:18.470590 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.470356    2581 scope.go:117] "RemoveContainer" containerID="37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f"
Sep 17 17:48:18.470938 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.470914    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 17:48:18.479634 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.479619    2581 scope.go:117] "RemoveContainer" containerID="89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28"
Sep 17 17:48:18.487774 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.487754    2581 scope.go:117] "RemoveContainer" containerID="37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f"
Sep 17 17:48:18.488041 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:48:18.488024    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f\": container with ID starting with 37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f not found: ID does not exist" containerID="37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f"
Sep 17 17:48:18.488116 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.488054    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f"} err="failed to get container status \"37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f\": rpc error: code = NotFound desc = could not find container \"37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f\": container with ID starting with 37a629b6041ee9f9745b8cc26949b6b8482a157ef72ba82457c8041721fc4b8f not found: ID does not exist"
Sep 17 17:48:18.488116 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.488075    2581 scope.go:117] "RemoveContainer" containerID="89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28"
Sep 17 17:48:18.488337 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:48:18.488320    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28\": container with ID starting with 89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28 not found: ID does not exist" containerID="89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28"
Sep 17 17:48:18.488393 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.488345    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28"} err="failed to get container status \"89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28\": rpc error: code = NotFound desc = could not find container \"89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28\": container with ID starting with 89ba626db33e35a7f91ed6852e26a3ddf456b197e0df5229c5877055faa21c28 not found: ID does not exist"
Sep 17 17:48:18.511125 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.511102    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"]
Sep 17 17:48:18.512769 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.512749    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a129-predictor-579fd87984-zc225"]
Sep 17 17:48:18.898756 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:18.898726    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" path="/var/lib/kubelet/pods/209300bb-fba8-4b66-95fb-77b9d25418e4/volumes"
Sep 17 17:48:23.422678 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:23.422633    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:23.475377 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:23.475351    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:48:23.475820 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:23.475791    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 17:48:28.421723 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:28.421662    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:28.422223 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:28.421777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:33.422316 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:33.422276    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:33.476651 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:33.476617    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 17:48:38.422099 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:38.422061    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:43.421726 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:43.421675    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:43.475787 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:43.475754    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 17:48:45.600072 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.600041    2581 generic.go:356] "Generic (PLEG): container finished" podID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerID="513c098f7407726f0260772ba5660587a848e6cfe9f26ea54aebfe2bd3147050" exitCode=0
Sep 17 17:48:45.600450 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.600111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" event={"ID":"b9fbc618-64f3-4112-9eb7-a54419edbb5e","Type":"ContainerDied","Data":"513c098f7407726f0260772ba5660587a848e6cfe9f26ea54aebfe2bd3147050"}
Sep 17 17:48:45.822040 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.822014    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:45.863207 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.863128    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9fbc618-64f3-4112-9eb7-a54419edbb5e-openshift-service-ca-bundle\") pod \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\" (UID: \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\") "
Sep 17 17:48:45.863580 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.863231    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9fbc618-64f3-4112-9eb7-a54419edbb5e-proxy-tls\") pod \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\" (UID: \"b9fbc618-64f3-4112-9eb7-a54419edbb5e\") "
Sep 17 17:48:45.863735 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.863568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fbc618-64f3-4112-9eb7-a54419edbb5e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b9fbc618-64f3-4112-9eb7-a54419edbb5e" (UID: "b9fbc618-64f3-4112-9eb7-a54419edbb5e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:48:45.865269 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.865249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9fbc618-64f3-4112-9eb7-a54419edbb5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9fbc618-64f3-4112-9eb7-a54419edbb5e" (UID: "b9fbc618-64f3-4112-9eb7-a54419edbb5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:48:45.963937 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.963901    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9fbc618-64f3-4112-9eb7-a54419edbb5e-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:48:45.963937 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:45.963929    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9fbc618-64f3-4112-9eb7-a54419edbb5e-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:48:46.605733 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:46.605688    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"
Sep 17 17:48:46.605733 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:46.605685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz" event={"ID":"b9fbc618-64f3-4112-9eb7-a54419edbb5e","Type":"ContainerDied","Data":"16a84e35561a02c8bbee3fb2f698347a68fbf2b3317560c1323724128e3bb711"}
Sep 17 17:48:46.606226 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:46.605818    2581 scope.go:117] "RemoveContainer" containerID="513c098f7407726f0260772ba5660587a848e6cfe9f26ea54aebfe2bd3147050"
Sep 17 17:48:46.631362 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:46.631340    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"]
Sep 17 17:48:46.636735 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:46.636697    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a129-8fdcd5874-qjxjz"]
Sep 17 17:48:46.898616 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:46.898543    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" path="/var/lib/kubelet/pods/b9fbc618-64f3-4112-9eb7-a54419edbb5e/volumes"
Sep 17 17:48:53.476173 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:48:53.476135    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 17:49:03.476866 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:03.476828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:49:04.995885 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.995849    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"]
Sep 17 17:49:04.996270 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996186    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container"
Sep 17 17:49:04.996270 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996196    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container"
Sep 17 17:49:04.996270 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996206    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kube-rbac-proxy"
Sep 17 17:49:04.996270 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996212    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kube-rbac-proxy"
Sep 17 17:49:04.996270 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996218    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129"
Sep 17 17:49:04.996270 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996224    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129"
Sep 17 17:49:04.996486 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9fbc618-64f3-4112-9eb7-a54419edbb5e" containerName="switch-graph-4a129"
Sep 17 17:49:04.996486 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996308    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kserve-container"
Sep 17 17:49:04.996486 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.996315    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="209300bb-fba8-4b66-95fb-77b9d25418e4" containerName="kube-rbac-proxy"
Sep 17 17:49:04.999173 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:04.999152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.003676 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.003657    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 17 17:49:05.003814 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.003754    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 17 17:49:05.012372 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.012350    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"]
Sep 17 17:49:05.133297 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.133263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566e6261-8653-46d7-bb73-d11160a124cf-proxy-tls\") pod \"model-chainer-597fd9c774-rhswc\" (UID: \"566e6261-8653-46d7-bb73-d11160a124cf\") " pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.133509 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.133327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566e6261-8653-46d7-bb73-d11160a124cf-openshift-service-ca-bundle\") pod \"model-chainer-597fd9c774-rhswc\" (UID: \"566e6261-8653-46d7-bb73-d11160a124cf\") " pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.234183 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.234144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566e6261-8653-46d7-bb73-d11160a124cf-proxy-tls\") pod \"model-chainer-597fd9c774-rhswc\" (UID: \"566e6261-8653-46d7-bb73-d11160a124cf\") " pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.234356 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.234204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566e6261-8653-46d7-bb73-d11160a124cf-openshift-service-ca-bundle\") pod \"model-chainer-597fd9c774-rhswc\" (UID: \"566e6261-8653-46d7-bb73-d11160a124cf\") " pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.234806 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.234789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566e6261-8653-46d7-bb73-d11160a124cf-openshift-service-ca-bundle\") pod \"model-chainer-597fd9c774-rhswc\" (UID: \"566e6261-8653-46d7-bb73-d11160a124cf\") " pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.236569 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.236550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566e6261-8653-46d7-bb73-d11160a124cf-proxy-tls\") pod \"model-chainer-597fd9c774-rhswc\" (UID: \"566e6261-8653-46d7-bb73-d11160a124cf\") " pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.309327 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.309253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.594309 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.594281    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"]
Sep 17 17:49:05.596845 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:49:05.596818    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566e6261_8653_46d7_bb73_d11160a124cf.slice/crio-96248a7cea4c973514381649987c058b44eea0a4d22b49edf1a33b354d32c4c5 WatchSource:0}: Error finding container 96248a7cea4c973514381649987c058b44eea0a4d22b49edf1a33b354d32c4c5: Status 404 returned error can't find the container with id 96248a7cea4c973514381649987c058b44eea0a4d22b49edf1a33b354d32c4c5
Sep 17 17:49:05.695134 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.695100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" event={"ID":"566e6261-8653-46d7-bb73-d11160a124cf","Type":"ContainerStarted","Data":"eec206672bd11385ffd6d8c53b28f102a4ee7fdf05ebe7d92ce419a8e5205389"}
Sep 17 17:49:05.695444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.695149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" event={"ID":"566e6261-8653-46d7-bb73-d11160a124cf","Type":"ContainerStarted","Data":"96248a7cea4c973514381649987c058b44eea0a4d22b49edf1a33b354d32c4c5"}
Sep 17 17:49:05.695444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.695180    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:05.720660 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:05.720610    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" podStartSLOduration=1.720596301 podStartE2EDuration="1.720596301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:05.718296976 +0000 UTC m=+860.950327796" watchObservedRunningTime="2026-09-17 17:49:05.720596301 +0000 UTC m=+860.952627119"
Sep 17 17:49:11.705549 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:11.705510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:15.481058 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.481026    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"]
Sep 17 17:49:15.484775 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.484757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:15.487779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.487753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8142b-kube-rbac-proxy-sar-config\""
Sep 17 17:49:15.487904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.487837    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8142b-serving-cert\""
Sep 17 17:49:15.507366 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.507342    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"]
Sep 17 17:49:15.611607 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.611585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca04026-4b27-41c0-bf37-a4fdde4143ce-proxy-tls\") pod \"switch-graph-8142b-d9d5fcc5c-79qkv\" (UID: \"fca04026-4b27-41c0-bf37-a4fdde4143ce\") " pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:15.611736 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.611637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca04026-4b27-41c0-bf37-a4fdde4143ce-openshift-service-ca-bundle\") pod \"switch-graph-8142b-d9d5fcc5c-79qkv\" (UID: \"fca04026-4b27-41c0-bf37-a4fdde4143ce\") " pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:15.712235 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.712205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca04026-4b27-41c0-bf37-a4fdde4143ce-openshift-service-ca-bundle\") pod \"switch-graph-8142b-d9d5fcc5c-79qkv\" (UID: \"fca04026-4b27-41c0-bf37-a4fdde4143ce\") " pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:15.712461 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.712444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca04026-4b27-41c0-bf37-a4fdde4143ce-proxy-tls\") pod \"switch-graph-8142b-d9d5fcc5c-79qkv\" (UID: \"fca04026-4b27-41c0-bf37-a4fdde4143ce\") " pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:15.713059 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.713037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca04026-4b27-41c0-bf37-a4fdde4143ce-openshift-service-ca-bundle\") pod \"switch-graph-8142b-d9d5fcc5c-79qkv\" (UID: \"fca04026-4b27-41c0-bf37-a4fdde4143ce\") " pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:15.715113 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.715089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca04026-4b27-41c0-bf37-a4fdde4143ce-proxy-tls\") pod \"switch-graph-8142b-d9d5fcc5c-79qkv\" (UID: \"fca04026-4b27-41c0-bf37-a4fdde4143ce\") " pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:15.754139 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.754115    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"]
Sep 17 17:49:15.754336 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.754295    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer" containerID="cri-o://eec206672bd11385ffd6d8c53b28f102a4ee7fdf05ebe7d92ce419a8e5205389" gracePeriod=30
Sep 17 17:49:15.795035 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:15.795014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:16.114904 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.114832    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"]
Sep 17 17:49:16.118451 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:49:16.118427    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca04026_4b27_41c0_bf37_a4fdde4143ce.slice/crio-071689184b2df773d794622acc7e8a18d50e6f3319488f2b50bc1ba364db96f2 WatchSource:0}: Error finding container 071689184b2df773d794622acc7e8a18d50e6f3319488f2b50bc1ba364db96f2: Status 404 returned error can't find the container with id 071689184b2df773d794622acc7e8a18d50e6f3319488f2b50bc1ba364db96f2
Sep 17 17:49:16.431313 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.431242    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"]
Sep 17 17:49:16.434946 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.434928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.441079 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.440873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dcaf3-predictor-serving-cert\""
Sep 17 17:49:16.441979 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.441958    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dcaf3-kube-rbac-proxy-sar-config\""
Sep 17 17:49:16.464015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.463993    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"]
Sep 17 17:49:16.520442 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.520419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4fp\" (UniqueName: \"kubernetes.io/projected/6c84b656-db79-4375-ab76-87850b55e7b1-kube-api-access-tp4fp\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.520743 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.520490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c84b656-db79-4375-ab76-87850b55e7b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.520743 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.520511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c84b656-db79-4375-ab76-87850b55e7b1-proxy-tls\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.621016 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.620992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c84b656-db79-4375-ab76-87850b55e7b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.621119 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.621020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c84b656-db79-4375-ab76-87850b55e7b1-proxy-tls\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.621119 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.621045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4fp\" (UniqueName: \"kubernetes.io/projected/6c84b656-db79-4375-ab76-87850b55e7b1-kube-api-access-tp4fp\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.621678 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.621657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c84b656-db79-4375-ab76-87850b55e7b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.623378 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.623356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c84b656-db79-4375-ab76-87850b55e7b1-proxy-tls\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.634115 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.634096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4fp\" (UniqueName: \"kubernetes.io/projected/6c84b656-db79-4375-ab76-87850b55e7b1-kube-api-access-tp4fp\") pod \"error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.704491 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.704431    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:16.744953 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.744926    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:16.752751 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.752728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" event={"ID":"fca04026-4b27-41c0-bf37-a4fdde4143ce","Type":"ContainerStarted","Data":"d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd"}
Sep 17 17:49:16.752837 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.752774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" event={"ID":"fca04026-4b27-41c0-bf37-a4fdde4143ce","Type":"ContainerStarted","Data":"071689184b2df773d794622acc7e8a18d50e6f3319488f2b50bc1ba364db96f2"}
Sep 17 17:49:16.752837 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:16.752816    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:17.079208 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.079163    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" podStartSLOduration=2.079145711 podStartE2EDuration="2.079145711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:16.789957542 +0000 UTC m=+872.021988366" watchObservedRunningTime="2026-09-17 17:49:17.079145711 +0000 UTC m=+872.311176530"
Sep 17 17:49:17.079941 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.079924    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"]
Sep 17 17:49:17.082577 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:49:17.082553    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c84b656_db79_4375_ab76_87850b55e7b1.slice/crio-4155ee64b1d683cc21a13b78ebe2b060bcf49ea4308d2b0e95f5f79b2e833e9b WatchSource:0}: Error finding container 4155ee64b1d683cc21a13b78ebe2b060bcf49ea4308d2b0e95f5f79b2e833e9b: Status 404 returned error can't find the container with id 4155ee64b1d683cc21a13b78ebe2b060bcf49ea4308d2b0e95f5f79b2e833e9b
Sep 17 17:49:17.758519 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.758487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" event={"ID":"6c84b656-db79-4375-ab76-87850b55e7b1","Type":"ContainerStarted","Data":"68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1"}
Sep 17 17:49:17.758923 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.758548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" event={"ID":"6c84b656-db79-4375-ab76-87850b55e7b1","Type":"ContainerStarted","Data":"6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099"}
Sep 17 17:49:17.758923 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.758568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" event={"ID":"6c84b656-db79-4375-ab76-87850b55e7b1","Type":"ContainerStarted","Data":"4155ee64b1d683cc21a13b78ebe2b060bcf49ea4308d2b0e95f5f79b2e833e9b"}
Sep 17 17:49:17.758923 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.758595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:17.758923 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.758624    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:17.759968 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.759946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 17:49:17.790996 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:17.790956    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podStartSLOduration=1.790942182 podStartE2EDuration="1.790942182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:17.789850469 +0000 UTC m=+873.021881290" watchObservedRunningTime="2026-09-17 17:49:17.790942182 +0000 UTC m=+873.022973001"
Sep 17 17:49:18.763494 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:18.763458    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 17:49:21.704145 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:21.704104    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:22.764161 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:22.764131    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:49:23.768180 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:23.768151    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:49:23.768647 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:23.768516    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 17:49:26.704249 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:26.704212    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:26.704619 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:26.704308    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:31.704462 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:31.704373    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:33.769322 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:33.769280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 17:49:36.704773 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:36.704730    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:41.704321 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:41.704281    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:43.769307 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:43.769264    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 17:49:45.890669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.890641    2581 generic.go:356] "Generic (PLEG): container finished" podID="566e6261-8653-46d7-bb73-d11160a124cf" containerID="eec206672bd11385ffd6d8c53b28f102a4ee7fdf05ebe7d92ce419a8e5205389" exitCode=0
Sep 17 17:49:45.890968 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.890723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" event={"ID":"566e6261-8653-46d7-bb73-d11160a124cf","Type":"ContainerDied","Data":"eec206672bd11385ffd6d8c53b28f102a4ee7fdf05ebe7d92ce419a8e5205389"}
Sep 17 17:49:45.890968 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.890780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc" event={"ID":"566e6261-8653-46d7-bb73-d11160a124cf","Type":"ContainerDied","Data":"96248a7cea4c973514381649987c058b44eea0a4d22b49edf1a33b354d32c4c5"}
Sep 17 17:49:45.890968 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.890796    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96248a7cea4c973514381649987c058b44eea0a4d22b49edf1a33b354d32c4c5"
Sep 17 17:49:45.905158 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.905137    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:45.974031 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.974003    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566e6261-8653-46d7-bb73-d11160a124cf-proxy-tls\") pod \"566e6261-8653-46d7-bb73-d11160a124cf\" (UID: \"566e6261-8653-46d7-bb73-d11160a124cf\") "
Sep 17 17:49:45.974192 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.974046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566e6261-8653-46d7-bb73-d11160a124cf-openshift-service-ca-bundle\") pod \"566e6261-8653-46d7-bb73-d11160a124cf\" (UID: \"566e6261-8653-46d7-bb73-d11160a124cf\") "
Sep 17 17:49:45.974495 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.974467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566e6261-8653-46d7-bb73-d11160a124cf-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "566e6261-8653-46d7-bb73-d11160a124cf" (UID: "566e6261-8653-46d7-bb73-d11160a124cf"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:49:45.976257 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:45.976239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566e6261-8653-46d7-bb73-d11160a124cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "566e6261-8653-46d7-bb73-d11160a124cf" (UID: "566e6261-8653-46d7-bb73-d11160a124cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:49:46.075503 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:46.075419    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566e6261-8653-46d7-bb73-d11160a124cf-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:49:46.075503 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:46.075449    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566e6261-8653-46d7-bb73-d11160a124cf-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:49:46.815146 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:46.815121    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:49:46.817366 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:46.817345    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:49:46.895259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:46.895221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"
Sep 17 17:49:46.931940 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:46.931904    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"]
Sep 17 17:49:46.940241 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:46.940218    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-597fd9c774-rhswc"]
Sep 17 17:49:48.899163 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:48.899132    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566e6261-8653-46d7-bb73-d11160a124cf" path="/var/lib/kubelet/pods/566e6261-8653-46d7-bb73-d11160a124cf/volumes"
Sep 17 17:49:53.769211 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:49:53.769169    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 17:50:03.769856 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:03.769822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:50:16.006906 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.006864    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"]
Sep 17 17:50:16.007394 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.007264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer"
Sep 17 17:50:16.007394 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.007277    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer"
Sep 17 17:50:16.007394 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.007354    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="566e6261-8653-46d7-bb73-d11160a124cf" containerName="model-chainer"
Sep 17 17:50:16.010288 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.010270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:16.013907 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.013889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-dcaf3-serving-cert\""
Sep 17 17:50:16.014595 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.014579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-dcaf3-kube-rbac-proxy-sar-config\""
Sep 17 17:50:16.026963 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.026942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/654d0db9-d499-45b7-b36b-8b011aed4640-proxy-tls\") pod \"sequence-graph-dcaf3-8fd969546-vzrkj\" (UID: \"654d0db9-d499-45b7-b36b-8b011aed4640\") " pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:16.027056 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.027010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/654d0db9-d499-45b7-b36b-8b011aed4640-openshift-service-ca-bundle\") pod \"sequence-graph-dcaf3-8fd969546-vzrkj\" (UID: \"654d0db9-d499-45b7-b36b-8b011aed4640\") " pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:16.036489 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.036466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"]
Sep 17 17:50:16.127910 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.127878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/654d0db9-d499-45b7-b36b-8b011aed4640-openshift-service-ca-bundle\") pod \"sequence-graph-dcaf3-8fd969546-vzrkj\" (UID: \"654d0db9-d499-45b7-b36b-8b011aed4640\") " pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:16.128074 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.127945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/654d0db9-d499-45b7-b36b-8b011aed4640-proxy-tls\") pod \"sequence-graph-dcaf3-8fd969546-vzrkj\" (UID: \"654d0db9-d499-45b7-b36b-8b011aed4640\") " pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:16.128481 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.128459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/654d0db9-d499-45b7-b36b-8b011aed4640-openshift-service-ca-bundle\") pod \"sequence-graph-dcaf3-8fd969546-vzrkj\" (UID: \"654d0db9-d499-45b7-b36b-8b011aed4640\") " pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:16.130244 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.130222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/654d0db9-d499-45b7-b36b-8b011aed4640-proxy-tls\") pod \"sequence-graph-dcaf3-8fd969546-vzrkj\" (UID: \"654d0db9-d499-45b7-b36b-8b011aed4640\") " pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:16.320558 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.320473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:16.620140 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:16.620113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"]
Sep 17 17:50:16.622750 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:50:16.622724    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654d0db9_d499_45b7_b36b_8b011aed4640.slice/crio-0f8cbea59ded75991789494599442e998bdb8d854a44fa1b36540e70f2ef60ad WatchSource:0}: Error finding container 0f8cbea59ded75991789494599442e998bdb8d854a44fa1b36540e70f2ef60ad: Status 404 returned error can't find the container with id 0f8cbea59ded75991789494599442e998bdb8d854a44fa1b36540e70f2ef60ad
Sep 17 17:50:17.042196 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:17.042152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" event={"ID":"654d0db9-d499-45b7-b36b-8b011aed4640","Type":"ContainerStarted","Data":"6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1"}
Sep 17 17:50:17.042659 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:17.042263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" event={"ID":"654d0db9-d499-45b7-b36b-8b011aed4640","Type":"ContainerStarted","Data":"0f8cbea59ded75991789494599442e998bdb8d854a44fa1b36540e70f2ef60ad"}
Sep 17 17:50:17.042855 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:17.042832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:50:17.085424 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:17.085376    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" podStartSLOduration=2.085362146 podStartE2EDuration="2.085362146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:17.082776538 +0000 UTC m=+932.314807356" watchObservedRunningTime="2026-09-17 17:50:17.085362146 +0000 UTC m=+932.317392964"
Sep 17 17:50:24.057423 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:50:24.057389    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:54:46.847678 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:54:46.847652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:54:46.848224 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:54:46.847741    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:55:46.891954 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:55:46.891879    2581 scope.go:117] "RemoveContainer" containerID="eec206672bd11385ffd6d8c53b28f102a4ee7fdf05ebe7d92ce419a8e5205389"
Sep 17 17:57:29.979966 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:29.979931    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"]
Sep 17 17:57:29.980536 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:29.980191    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b" containerID="cri-o://d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd" gracePeriod=30
Sep 17 17:57:30.297977 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.297944    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"]
Sep 17 17:57:30.298177 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.298154    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container" containerID="cri-o://e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441" gracePeriod=30
Sep 17 17:57:30.298296 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.298243    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kube-rbac-proxy" containerID="cri-o://e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0" gracePeriod=30
Sep 17 17:57:30.318738 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:57:30.318685    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2409f72_3a11_4c01_b722_8b130754e5ae.slice/crio-e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 17:57:30.377859 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.377823    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"]
Sep 17 17:57:30.381557 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.381535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:30.384655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.384634    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1fda1-predictor-serving-cert\""
Sep 17 17:57:30.384655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.384650    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1fda1-kube-rbac-proxy-sar-config\""
Sep 17 17:57:30.402338 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.402316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"]
Sep 17 17:57:30.514479 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.514448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:30.514641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.514488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:30.514641 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.514514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4cwz\" (UniqueName: \"kubernetes.io/projected/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-api-access-f4cwz\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:30.616083 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.616001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:30.616083 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.616038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:30.616083 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.616063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4cwz\" (UniqueName: \"kubernetes.io/projected/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-api-access-f4cwz\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:30.616335 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:57:30.616153    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-serving-cert: secret "error-404-isvc-1fda1-predictor-serving-cert" not found
Sep 17 17:57:30.616335 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:57:30.616226    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls podName:e4cba543-f1b7-47ad-b3bc-0d1eac81f333 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:31.116205781 +0000 UTC m=+1366.348236578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls") pod "error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" (UID: "e4cba543-f1b7-47ad-b3bc-0d1eac81f333") : secret "error-404-isvc-1fda1-predictor-serving-cert" not found
Sep 17 17:57:30.616625 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.616606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:30.626537 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:30.626510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4cwz\" (UniqueName: \"kubernetes.io/projected/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-api-access-f4cwz\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:31.112180 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:31.112142    2581 generic.go:356] "Generic (PLEG): container finished" podID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerID="e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0" exitCode=2
Sep 17 17:57:31.112559 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:31.112215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" event={"ID":"e2409f72-3a11-4c01-b722-8b130754e5ae","Type":"ContainerDied","Data":"e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0"}
Sep 17 17:57:31.120607 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:31.120586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:31.122962 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:31.122944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls\") pod \"error-404-isvc-1fda1-predictor-c86ccb888-n4lkk\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") " pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:31.291620 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:31.291578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:31.571221 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:31.571188    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"]
Sep 17 17:57:31.574555 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:57:31.574525    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cba543_f1b7_47ad_b3bc_0d1eac81f333.slice/crio-e461feb42d0d1bfb31ce17c9e5bec6da8b853f2d25b0770c87f55f05b529cfcb WatchSource:0}: Error finding container e461feb42d0d1bfb31ce17c9e5bec6da8b853f2d25b0770c87f55f05b529cfcb: Status 404 returned error can't find the container with id e461feb42d0d1bfb31ce17c9e5bec6da8b853f2d25b0770c87f55f05b529cfcb
Sep 17 17:57:32.118100 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:32.118066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" event={"ID":"e4cba543-f1b7-47ad-b3bc-0d1eac81f333","Type":"ContainerStarted","Data":"802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2"}
Sep 17 17:57:32.118524 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:32.118128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" event={"ID":"e4cba543-f1b7-47ad-b3bc-0d1eac81f333","Type":"ContainerStarted","Data":"766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a"}
Sep 17 17:57:32.118524 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:32.118241    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:32.118524 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:32.118381    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:32.118524 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:32.118443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" event={"ID":"e4cba543-f1b7-47ad-b3bc-0d1eac81f333","Type":"ContainerStarted","Data":"e461feb42d0d1bfb31ce17c9e5bec6da8b853f2d25b0770c87f55f05b529cfcb"}
Sep 17 17:57:32.119761 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:32.119735    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 17:57:32.145996 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:32.145943    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podStartSLOduration=2.145926554 podStartE2EDuration="2.145926554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:32.144215955 +0000 UTC m=+1367.376246798" watchObservedRunningTime="2026-09-17 17:57:32.145926554 +0000 UTC m=+1367.377957372"
Sep 17 17:57:32.763416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:32.763373    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:33.122623 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.122537    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 17:57:33.471321 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.471288    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 17:57:33.476628 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.476605    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 17:57:33.662486 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.662463    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:57:33.845337 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.845309    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxqkq\" (UniqueName: \"kubernetes.io/projected/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-api-access-wxqkq\") pod \"e2409f72-3a11-4c01-b722-8b130754e5ae\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") "
Sep 17 17:57:33.845510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.845350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2409f72-3a11-4c01-b722-8b130754e5ae-proxy-tls\") pod \"e2409f72-3a11-4c01-b722-8b130754e5ae\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") "
Sep 17 17:57:33.845510 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.845425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-rbac-proxy-sar-config\") pod \"e2409f72-3a11-4c01-b722-8b130754e5ae\" (UID: \"e2409f72-3a11-4c01-b722-8b130754e5ae\") "
Sep 17 17:57:33.852952 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.846458    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2409f72-3a11-4c01-b722-8b130754e5ae" (UID: "e2409f72-3a11-4c01-b722-8b130754e5ae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:33.853465 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.853332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2409f72-3a11-4c01-b722-8b130754e5ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2409f72-3a11-4c01-b722-8b130754e5ae" (UID: "e2409f72-3a11-4c01-b722-8b130754e5ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:33.853567 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.853460    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-api-access-wxqkq" (OuterVolumeSpecName: "kube-api-access-wxqkq") pod "e2409f72-3a11-4c01-b722-8b130754e5ae" (UID: "e2409f72-3a11-4c01-b722-8b130754e5ae"). InnerVolumeSpecName "kube-api-access-wxqkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:33.946508 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.946477    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2409f72-3a11-4c01-b722-8b130754e5ae-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:57:33.946508 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.946505    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:57:33.946714 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:33.946516    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqkq\" (UniqueName: \"kubernetes.io/projected/e2409f72-3a11-4c01-b722-8b130754e5ae-kube-api-access-wxqkq\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:57:34.129152 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.129059    2581 generic.go:356] "Generic (PLEG): container finished" podID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerID="e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441" exitCode=0
Sep 17 17:57:34.129152 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.129121    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"
Sep 17 17:57:34.129669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.129143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" event={"ID":"e2409f72-3a11-4c01-b722-8b130754e5ae","Type":"ContainerDied","Data":"e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441"}
Sep 17 17:57:34.129669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.129214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng" event={"ID":"e2409f72-3a11-4c01-b722-8b130754e5ae","Type":"ContainerDied","Data":"5751d1aa4259a577399909505b2a7319001143c24ea4f56b26e23e50a2b0390b"}
Sep 17 17:57:34.129669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.129254    2581 scope.go:117] "RemoveContainer" containerID="e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0"
Sep 17 17:57:34.138496 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.138457    2581 scope.go:117] "RemoveContainer" containerID="e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441"
Sep 17 17:57:34.146597 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.146578    2581 scope.go:117] "RemoveContainer" containerID="e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0"
Sep 17 17:57:34.146899 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:57:34.146875    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0\": container with ID starting with e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0 not found: ID does not exist" containerID="e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0"
Sep 17 17:57:34.146955 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.146908    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0"} err="failed to get container status \"e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0\": rpc error: code = NotFound desc = could not find container \"e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0\": container with ID starting with e89b81079cdad3dbf2ff684b0d0f332714cc505543ecca1d641f1b8b3cbde2b0 not found: ID does not exist"
Sep 17 17:57:34.146955 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.146924    2581 scope.go:117] "RemoveContainer" containerID="e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441"
Sep 17 17:57:34.147154 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:57:34.147139    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441\": container with ID starting with e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441 not found: ID does not exist" containerID="e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441"
Sep 17 17:57:34.147207 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.147157    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441"} err="failed to get container status \"e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441\": rpc error: code = NotFound desc = could not find container \"e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441\": container with ID starting with e10956c6cacca8769be42691dc2178c2b8e3943c52450aed86bbaad306482441 not found: ID does not exist"
Sep 17 17:57:34.165844 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.165824    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"]
Sep 17 17:57:34.170797 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.170777    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8142b-predictor-7948b5d7c6-zd5ng"]
Sep 17 17:57:34.898947 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:34.898917    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" path="/var/lib/kubelet/pods/e2409f72-3a11-4c01-b722-8b130754e5ae/volumes"
Sep 17 17:57:37.763125 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:37.763085    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:38.127783 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:38.127690    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:57:38.128106 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:38.128081    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 17:57:42.762392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:42.762352    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:42.762778 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:42.762450    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:57:47.762863 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:47.762825    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:48.128848 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:48.128758    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 17:57:52.763439 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:52.763402    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:57.763205 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:57.763163    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:57:58.128390 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:57:58.128305    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 17:58:00.128444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.128417    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:58:00.151357 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.151334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca04026-4b27-41c0-bf37-a4fdde4143ce-proxy-tls\") pod \"fca04026-4b27-41c0-bf37-a4fdde4143ce\" (UID: \"fca04026-4b27-41c0-bf37-a4fdde4143ce\") "
Sep 17 17:58:00.151541 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.151522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca04026-4b27-41c0-bf37-a4fdde4143ce-openshift-service-ca-bundle\") pod \"fca04026-4b27-41c0-bf37-a4fdde4143ce\" (UID: \"fca04026-4b27-41c0-bf37-a4fdde4143ce\") "
Sep 17 17:58:00.151855 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.151833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca04026-4b27-41c0-bf37-a4fdde4143ce-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fca04026-4b27-41c0-bf37-a4fdde4143ce" (UID: "fca04026-4b27-41c0-bf37-a4fdde4143ce"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:00.153286 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.153254    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca04026-4b27-41c0-bf37-a4fdde4143ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fca04026-4b27-41c0-bf37-a4fdde4143ce" (UID: "fca04026-4b27-41c0-bf37-a4fdde4143ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:00.252367 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.252344    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca04026-4b27-41c0-bf37-a4fdde4143ce-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:58:00.252367 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.252366    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fca04026-4b27-41c0-bf37-a4fdde4143ce-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:58:00.257886 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.257861    2581 generic.go:356] "Generic (PLEG): container finished" podID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerID="d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd" exitCode=0
Sep 17 17:58:00.257999 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.257903    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"
Sep 17 17:58:00.257999 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.257945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" event={"ID":"fca04026-4b27-41c0-bf37-a4fdde4143ce","Type":"ContainerDied","Data":"d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd"}
Sep 17 17:58:00.258093 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.258017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv" event={"ID":"fca04026-4b27-41c0-bf37-a4fdde4143ce","Type":"ContainerDied","Data":"071689184b2df773d794622acc7e8a18d50e6f3319488f2b50bc1ba364db96f2"}
Sep 17 17:58:00.258093 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.258040    2581 scope.go:117] "RemoveContainer" containerID="d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd"
Sep 17 17:58:00.266392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.266375    2581 scope.go:117] "RemoveContainer" containerID="d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd"
Sep 17 17:58:00.266653 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:58:00.266636    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd\": container with ID starting with d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd not found: ID does not exist" containerID="d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd"
Sep 17 17:58:00.266769 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.266664    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd"} err="failed to get container status \"d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd\": rpc error: code = NotFound desc = could not find container \"d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd\": container with ID starting with d474b0568632fda41815f1845d273f8e9eeeb9c204719acab708b58a49fc0cdd not found: ID does not exist"
Sep 17 17:58:00.293879 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.293857    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"]
Sep 17 17:58:00.309911 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.309888    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8142b-d9d5fcc5c-79qkv"]
Sep 17 17:58:00.899101 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:00.899067    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" path="/var/lib/kubelet/pods/fca04026-4b27-41c0-bf37-a4fdde4143ce/volumes"
Sep 17 17:58:08.128480 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:08.128439    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 17:58:18.128859 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:18.128828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:58:30.353055 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.352945    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"]
Sep 17 17:58:30.353655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353525    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kube-rbac-proxy"
Sep 17 17:58:30.353655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353546    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kube-rbac-proxy"
Sep 17 17:58:30.353655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353563    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b"
Sep 17 17:58:30.353655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353572    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b"
Sep 17 17:58:30.353655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353602    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container"
Sep 17 17:58:30.353655 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353610    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container"
Sep 17 17:58:30.354000 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353738    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kserve-container"
Sep 17 17:58:30.354000 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353750    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fca04026-4b27-41c0-bf37-a4fdde4143ce" containerName="switch-graph-8142b"
Sep 17 17:58:30.354000 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.353759    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2409f72-3a11-4c01-b722-8b130754e5ae" containerName="kube-rbac-proxy"
Sep 17 17:58:30.356945 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.356927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:30.359859 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.359838    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-1fda1-kube-rbac-proxy-sar-config\""
Sep 17 17:58:30.360098 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.360085    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-1fda1-serving-cert\""
Sep 17 17:58:30.368644 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.368621    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"]
Sep 17 17:58:30.513377 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.513342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-proxy-tls\") pod \"ensemble-graph-1fda1-756f77b956-96xnd\" (UID: \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\") " pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:30.513533 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.513400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-openshift-service-ca-bundle\") pod \"ensemble-graph-1fda1-756f77b956-96xnd\" (UID: \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\") " pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:30.614133 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.614053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-proxy-tls\") pod \"ensemble-graph-1fda1-756f77b956-96xnd\" (UID: \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\") " pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:30.614133 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.614119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-openshift-service-ca-bundle\") pod \"ensemble-graph-1fda1-756f77b956-96xnd\" (UID: \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\") " pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:30.614717 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.614684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-openshift-service-ca-bundle\") pod \"ensemble-graph-1fda1-756f77b956-96xnd\" (UID: \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\") " pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:30.616477 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.616456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-proxy-tls\") pod \"ensemble-graph-1fda1-756f77b956-96xnd\" (UID: \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\") " pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:30.667281 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.667237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:30.722668 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.722516    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"]
Sep 17 17:58:30.722811 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.722781    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3" containerID="cri-o://6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1" gracePeriod=30
Sep 17 17:58:30.821680 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.821647    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"]
Sep 17 17:58:30.821919 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.821875    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container" containerID="cri-o://6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099" gracePeriod=30
Sep 17 17:58:30.821919 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.821903    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kube-rbac-proxy" containerID="cri-o://68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1" gracePeriod=30
Sep 17 17:58:30.969380 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:58:30.969347    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f6bdcd_2728_4385_bec1_9f4fc1bcdc36.slice/crio-f66d5e652405004dbcb31cbb9434905a6244330547d1feeef35db050d435e191 WatchSource:0}: Error finding container f66d5e652405004dbcb31cbb9434905a6244330547d1feeef35db050d435e191: Status 404 returned error can't find the container with id f66d5e652405004dbcb31cbb9434905a6244330547d1feeef35db050d435e191
Sep 17 17:58:30.969639 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:30.969614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"]
Sep 17 17:58:31.111629 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.111596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"]
Sep 17 17:58:31.115253 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.115236    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.118157 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.118136    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c397f-predictor-serving-cert\""
Sep 17 17:58:31.118425 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.118407    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c397f-kube-rbac-proxy-sar-config\""
Sep 17 17:58:31.135838 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.135815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"]
Sep 17 17:58:31.218187 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.218121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21b643b5-d440-4a78-8c63-f1870f0a20fd-proxy-tls\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.218187 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.218159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.218365 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.218264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvf9j\" (UniqueName: \"kubernetes.io/projected/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-api-access-dvf9j\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.318730 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.318674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf9j\" (UniqueName: \"kubernetes.io/projected/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-api-access-dvf9j\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.318875 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.318776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21b643b5-d440-4a78-8c63-f1870f0a20fd-proxy-tls\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.318875 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.318819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.319485 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.319462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.321136 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.321120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21b643b5-d440-4a78-8c63-f1870f0a20fd-proxy-tls\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.335147 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.335122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf9j\" (UniqueName: \"kubernetes.io/projected/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-api-access-dvf9j\") pod \"error-404-isvc-c397f-predictor-699fb4686-ws4j5\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") " pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.403982 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.403956    2581 generic.go:356] "Generic (PLEG): container finished" podID="6c84b656-db79-4375-ab76-87850b55e7b1" containerID="68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1" exitCode=2
Sep 17 17:58:31.404306 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.404036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" event={"ID":"6c84b656-db79-4375-ab76-87850b55e7b1","Type":"ContainerDied","Data":"68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1"}
Sep 17 17:58:31.405581 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.405562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" event={"ID":"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36","Type":"ContainerStarted","Data":"d708ca9a38f8e5b21b9cdbaadc1777193e017b394b9300c47a47c28cc3967955"}
Sep 17 17:58:31.405669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.405599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" event={"ID":"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36","Type":"ContainerStarted","Data":"f66d5e652405004dbcb31cbb9434905a6244330547d1feeef35db050d435e191"}
Sep 17 17:58:31.405669 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.405635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:31.424945 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.424922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:31.426063 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.426028    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" podStartSLOduration=1.426016317 podStartE2EDuration="1.426016317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:31.42484377 +0000 UTC m=+1426.656874588" watchObservedRunningTime="2026-09-17 17:58:31.426016317 +0000 UTC m=+1426.658047171"
Sep 17 17:58:31.700029 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:31.700003    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"]
Sep 17 17:58:31.702543 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:58:31.702518    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b643b5_d440_4a78_8c63_f1870f0a20fd.slice/crio-f53e226e4f5238c8fed574d076702633ac981c2cf720125ebd6c25079feccdd6 WatchSource:0}: Error finding container f53e226e4f5238c8fed574d076702633ac981c2cf720125ebd6c25079feccdd6: Status 404 returned error can't find the container with id f53e226e4f5238c8fed574d076702633ac981c2cf720125ebd6c25079feccdd6
Sep 17 17:58:32.413332 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:32.413293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" event={"ID":"21b643b5-d440-4a78-8c63-f1870f0a20fd","Type":"ContainerStarted","Data":"3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6"}
Sep 17 17:58:32.413896 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:32.413422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:32.413896 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:32.413483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" event={"ID":"21b643b5-d440-4a78-8c63-f1870f0a20fd","Type":"ContainerStarted","Data":"483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620"}
Sep 17 17:58:32.413896 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:32.413519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" event={"ID":"21b643b5-d440-4a78-8c63-f1870f0a20fd","Type":"ContainerStarted","Data":"f53e226e4f5238c8fed574d076702633ac981c2cf720125ebd6c25079feccdd6"}
Sep 17 17:58:32.413896 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:32.413726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:32.414735 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:32.414697    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 17:58:32.443779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:32.443730    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podStartSLOduration=1.443697699 podStartE2EDuration="1.443697699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:32.441055455 +0000 UTC m=+1427.673086276" watchObservedRunningTime="2026-09-17 17:58:32.443697699 +0000 UTC m=+1427.675728523"
Sep 17 17:58:33.419057 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:33.419018    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 17:58:33.764371 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:33.764335    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 17 17:58:33.768590 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:33.768560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 17:58:33.979246 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:33.979221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:58:34.055334 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.055249    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:34.145248 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.145222    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c84b656-db79-4375-ab76-87850b55e7b1-proxy-tls\") pod \"6c84b656-db79-4375-ab76-87850b55e7b1\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") "
Sep 17 17:58:34.145435 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.145327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c84b656-db79-4375-ab76-87850b55e7b1-kube-rbac-proxy-sar-config\") pod \"6c84b656-db79-4375-ab76-87850b55e7b1\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") "
Sep 17 17:58:34.145435 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.145349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp4fp\" (UniqueName: \"kubernetes.io/projected/6c84b656-db79-4375-ab76-87850b55e7b1-kube-api-access-tp4fp\") pod \"6c84b656-db79-4375-ab76-87850b55e7b1\" (UID: \"6c84b656-db79-4375-ab76-87850b55e7b1\") "
Sep 17 17:58:34.145664 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.145640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c84b656-db79-4375-ab76-87850b55e7b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c84b656-db79-4375-ab76-87850b55e7b1" (UID: "6c84b656-db79-4375-ab76-87850b55e7b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:34.147200 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.147168    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c84b656-db79-4375-ab76-87850b55e7b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c84b656-db79-4375-ab76-87850b55e7b1" (UID: "6c84b656-db79-4375-ab76-87850b55e7b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:34.147299 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.147278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c84b656-db79-4375-ab76-87850b55e7b1-kube-api-access-tp4fp" (OuterVolumeSpecName: "kube-api-access-tp4fp") pod "6c84b656-db79-4375-ab76-87850b55e7b1" (UID: "6c84b656-db79-4375-ab76-87850b55e7b1"). InnerVolumeSpecName "kube-api-access-tp4fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:34.246606 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.246576    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c84b656-db79-4375-ab76-87850b55e7b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:58:34.246606 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.246607    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp4fp\" (UniqueName: \"kubernetes.io/projected/6c84b656-db79-4375-ab76-87850b55e7b1-kube-api-access-tp4fp\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:58:34.246817 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.246623    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c84b656-db79-4375-ab76-87850b55e7b1-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:58:34.425675 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.425600    2581 generic.go:356] "Generic (PLEG): container finished" podID="6c84b656-db79-4375-ab76-87850b55e7b1" containerID="6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099" exitCode=0
Sep 17 17:58:34.425675 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.425650    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"
Sep 17 17:58:34.425675 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.425669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" event={"ID":"6c84b656-db79-4375-ab76-87850b55e7b1","Type":"ContainerDied","Data":"6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099"}
Sep 17 17:58:34.426125 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.425737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl" event={"ID":"6c84b656-db79-4375-ab76-87850b55e7b1","Type":"ContainerDied","Data":"4155ee64b1d683cc21a13b78ebe2b060bcf49ea4308d2b0e95f5f79b2e833e9b"}
Sep 17 17:58:34.426125 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.425765    2581 scope.go:117] "RemoveContainer" containerID="68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1"
Sep 17 17:58:34.435106 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.435087    2581 scope.go:117] "RemoveContainer" containerID="6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099"
Sep 17 17:58:34.442921 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.442894    2581 scope.go:117] "RemoveContainer" containerID="68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1"
Sep 17 17:58:34.443205 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:58:34.443183    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1\": container with ID starting with 68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1 not found: ID does not exist" containerID="68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1"
Sep 17 17:58:34.443258 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.443218    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1"} err="failed to get container status \"68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1\": rpc error: code = NotFound desc = could not find container \"68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1\": container with ID starting with 68e39ddb4ab837a4f20c3ad1b5585038ac40e88a029c4b56c1702ec86c508dc1 not found: ID does not exist"
Sep 17 17:58:34.443258 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.443240    2581 scope.go:117] "RemoveContainer" containerID="6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099"
Sep 17 17:58:34.443530 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:58:34.443507    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099\": container with ID starting with 6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099 not found: ID does not exist" containerID="6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099"
Sep 17 17:58:34.443588 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.443539    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099"} err="failed to get container status \"6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099\": rpc error: code = NotFound desc = could not find container \"6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099\": container with ID starting with 6a217469a97749221e3affa02583f8fa06e6fdd7da6e91a24e54d39fc7a06099 not found: ID does not exist"
Sep 17 17:58:34.455076 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.455056    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"]
Sep 17 17:58:34.465023 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.465003    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dcaf3-predictor-5fdbd86d9b-xb9bl"]
Sep 17 17:58:34.898688 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:34.898653    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" path="/var/lib/kubelet/pods/6c84b656-db79-4375-ab76-87850b55e7b1/volumes"
Sep 17 17:58:37.419355 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:37.419326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:38.423808 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:38.423781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:58:38.424223 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:38.424178    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 17:58:39.055059 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:39.055024    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:44.055763 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:44.055699    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:44.056168 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:44.055841    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:58:44.495266 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:44.495230    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"]
Sep 17 17:58:44.495448 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:44.495425    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1" containerID="cri-o://d708ca9a38f8e5b21b9cdbaadc1777193e017b394b9300c47a47c28cc3967955" gracePeriod=30
Sep 17 17:58:44.728132 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:44.728095    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"]
Sep 17 17:58:44.728349 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:44.728322    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container" containerID="cri-o://766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a" gracePeriod=30
Sep 17 17:58:44.728413 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:44.728380    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kube-rbac-proxy" containerID="cri-o://802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2" gracePeriod=30
Sep 17 17:58:45.051309 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.051272    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"]
Sep 17 17:58:45.051660 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.051647    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kube-rbac-proxy"
Sep 17 17:58:45.051734 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.051662    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kube-rbac-proxy"
Sep 17 17:58:45.051734 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.051679    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container"
Sep 17 17:58:45.051734 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.051684    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container"
Sep 17 17:58:45.051850 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.051765    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kserve-container"
Sep 17 17:58:45.051850 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.051773    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c84b656-db79-4375-ab76-87850b55e7b1" containerName="kube-rbac-proxy"
Sep 17 17:58:45.054999 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.054982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.058924 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.058899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1175d-predictor-serving-cert\""
Sep 17 17:58:45.060524 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.060505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1175d-kube-rbac-proxy-sar-config\""
Sep 17 17:58:45.072108 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.072082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"]
Sep 17 17:58:45.141767 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.141680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d6cea0-bed8-4b87-a22e-a770be732246-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.141935 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.141785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d6cea0-bed8-4b87-a22e-a770be732246-proxy-tls\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.141935 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.141866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hl9b\" (UniqueName: \"kubernetes.io/projected/53d6cea0-bed8-4b87-a22e-a770be732246-kube-api-access-7hl9b\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.243201 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.243109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d6cea0-bed8-4b87-a22e-a770be732246-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.243379 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.243303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d6cea0-bed8-4b87-a22e-a770be732246-proxy-tls\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.243432 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.243408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hl9b\" (UniqueName: \"kubernetes.io/projected/53d6cea0-bed8-4b87-a22e-a770be732246-kube-api-access-7hl9b\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.243923 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.243900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d6cea0-bed8-4b87-a22e-a770be732246-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.245748 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.245726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d6cea0-bed8-4b87-a22e-a770be732246-proxy-tls\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.256352 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.256327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hl9b\" (UniqueName: \"kubernetes.io/projected/53d6cea0-bed8-4b87-a22e-a770be732246-kube-api-access-7hl9b\") pod \"error-404-isvc-1175d-predictor-6784b69858-d8x9r\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") " pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.364302 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.364265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:45.494165 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.494133    2581 generic.go:356] "Generic (PLEG): container finished" podID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerID="802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2" exitCode=2
Sep 17 17:58:45.494317 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.494203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" event={"ID":"e4cba543-f1b7-47ad-b3bc-0d1eac81f333","Type":"ContainerDied","Data":"802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2"}
Sep 17 17:58:45.647806 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:45.647720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"]
Sep 17 17:58:45.650301 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:58:45.650273    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d6cea0_bed8_4b87_a22e_a770be732246.slice/crio-16d9a1f922368d80b237d0ad5eff263143c415e3ee61d4133c75a15994855eee WatchSource:0}: Error finding container 16d9a1f922368d80b237d0ad5eff263143c415e3ee61d4133c75a15994855eee: Status 404 returned error can't find the container with id 16d9a1f922368d80b237d0ad5eff263143c415e3ee61d4133c75a15994855eee
Sep 17 17:58:46.501624 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:46.501590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" event={"ID":"53d6cea0-bed8-4b87-a22e-a770be732246","Type":"ContainerStarted","Data":"cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b"}
Sep 17 17:58:46.502046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:46.501651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" event={"ID":"53d6cea0-bed8-4b87-a22e-a770be732246","Type":"ContainerStarted","Data":"4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137"}
Sep 17 17:58:46.502046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:46.501672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" event={"ID":"53d6cea0-bed8-4b87-a22e-a770be732246","Type":"ContainerStarted","Data":"16d9a1f922368d80b237d0ad5eff263143c415e3ee61d4133c75a15994855eee"}
Sep 17 17:58:46.502046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:46.501698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:46.502046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:46.501838    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:46.503084 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:46.503054    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 17:58:46.544874 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:46.544826    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podStartSLOduration=1.544809759 podStartE2EDuration="1.544809759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:46.544092225 +0000 UTC m=+1441.776123043" watchObservedRunningTime="2026-09-17 17:58:46.544809759 +0000 UTC m=+1441.776840579"
Sep 17 17:58:47.417820 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:47.417782    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:47.506967 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:47.506928    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 17:58:48.073648 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.073626    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:58:48.163673 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.163578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls\") pod \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") "
Sep 17 17:58:48.163673 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.163641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4cwz\" (UniqueName: \"kubernetes.io/projected/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-api-access-f4cwz\") pod \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") "
Sep 17 17:58:48.163893 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.163725    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-rbac-proxy-sar-config\") pod \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\" (UID: \"e4cba543-f1b7-47ad-b3bc-0d1eac81f333\") "
Sep 17 17:58:48.164083 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.164050    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4cba543-f1b7-47ad-b3bc-0d1eac81f333" (UID: "e4cba543-f1b7-47ad-b3bc-0d1eac81f333"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:48.165773 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.165739    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-api-access-f4cwz" (OuterVolumeSpecName: "kube-api-access-f4cwz") pod "e4cba543-f1b7-47ad-b3bc-0d1eac81f333" (UID: "e4cba543-f1b7-47ad-b3bc-0d1eac81f333"). InnerVolumeSpecName "kube-api-access-f4cwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:48.165773 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.165751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4cba543-f1b7-47ad-b3bc-0d1eac81f333" (UID: "e4cba543-f1b7-47ad-b3bc-0d1eac81f333"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:48.264819 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.264774    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.264819 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.264815    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.264819 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.264826    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4cwz\" (UniqueName: \"kubernetes.io/projected/e4cba543-f1b7-47ad-b3bc-0d1eac81f333-kube-api-access-f4cwz\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:58:48.424395 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.424298    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 17:58:48.512779 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.512745    2581 generic.go:356] "Generic (PLEG): container finished" podID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerID="766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a" exitCode=0
Sep 17 17:58:48.513187 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.512791    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"
Sep 17 17:58:48.513187 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.512829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" event={"ID":"e4cba543-f1b7-47ad-b3bc-0d1eac81f333","Type":"ContainerDied","Data":"766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a"}
Sep 17 17:58:48.513187 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.512892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk" event={"ID":"e4cba543-f1b7-47ad-b3bc-0d1eac81f333","Type":"ContainerDied","Data":"e461feb42d0d1bfb31ce17c9e5bec6da8b853f2d25b0770c87f55f05b529cfcb"}
Sep 17 17:58:48.513187 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.512920    2581 scope.go:117] "RemoveContainer" containerID="802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2"
Sep 17 17:58:48.521932 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.521914    2581 scope.go:117] "RemoveContainer" containerID="766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a"
Sep 17 17:58:48.530137 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.530119    2581 scope.go:117] "RemoveContainer" containerID="802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2"
Sep 17 17:58:48.530382 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:58:48.530364    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2\": container with ID starting with 802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2 not found: ID does not exist" containerID="802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2"
Sep 17 17:58:48.530444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.530389    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2"} err="failed to get container status \"802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2\": rpc error: code = NotFound desc = could not find container \"802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2\": container with ID starting with 802afe936a5fc5adfbd1f24d4e88482fe4edf2ab947861470aced0ef873d95f2 not found: ID does not exist"
Sep 17 17:58:48.530444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.530404    2581 scope.go:117] "RemoveContainer" containerID="766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a"
Sep 17 17:58:48.530633 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:58:48.530617    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a\": container with ID starting with 766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a not found: ID does not exist" containerID="766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a"
Sep 17 17:58:48.530673 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.530639    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a"} err="failed to get container status \"766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a\": rpc error: code = NotFound desc = could not find container \"766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a\": container with ID starting with 766a03d7c23adc5975229246c209c61f9499925999053c0f3d2bc11259bb070a not found: ID does not exist"
Sep 17 17:58:48.542932 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.542905    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"]
Sep 17 17:58:48.546326 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.546305    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fda1-predictor-c86ccb888-n4lkk"]
Sep 17 17:58:48.898483 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:48.898449    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" path="/var/lib/kubelet/pods/e4cba543-f1b7-47ad-b3bc-0d1eac81f333/volumes"
Sep 17 17:58:49.055119 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:49.055085    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:52.418046 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:52.418001    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:52.511774 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:52.511750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:58:52.512189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:52.512163    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 17:58:54.055298 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:54.055255    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:57.417401 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:57.417354    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:57.417771 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:57.417460    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:58:58.424795 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:58.424754    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 17:58:59.054826 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:58:59.054790    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:00.872667 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:00.872643    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:59:00.971647 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:00.971621    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/654d0db9-d499-45b7-b36b-8b011aed4640-openshift-service-ca-bundle\") pod \"654d0db9-d499-45b7-b36b-8b011aed4640\" (UID: \"654d0db9-d499-45b7-b36b-8b011aed4640\") "
Sep 17 17:59:00.971795 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:00.971741    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/654d0db9-d499-45b7-b36b-8b011aed4640-proxy-tls\") pod \"654d0db9-d499-45b7-b36b-8b011aed4640\" (UID: \"654d0db9-d499-45b7-b36b-8b011aed4640\") "
Sep 17 17:59:00.971955 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:00.971933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654d0db9-d499-45b7-b36b-8b011aed4640-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "654d0db9-d499-45b7-b36b-8b011aed4640" (UID: "654d0db9-d499-45b7-b36b-8b011aed4640"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:00.973625 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:00.973604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654d0db9-d499-45b7-b36b-8b011aed4640-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "654d0db9-d499-45b7-b36b-8b011aed4640" (UID: "654d0db9-d499-45b7-b36b-8b011aed4640"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:01.073240 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.073207    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/654d0db9-d499-45b7-b36b-8b011aed4640-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:59:01.073240 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.073239    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/654d0db9-d499-45b7-b36b-8b011aed4640-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:59:01.584513 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.584478    2581 generic.go:356] "Generic (PLEG): container finished" podID="654d0db9-d499-45b7-b36b-8b011aed4640" containerID="6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1" exitCode=0
Sep 17 17:59:01.584675 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.584522    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"
Sep 17 17:59:01.584675 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.584563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" event={"ID":"654d0db9-d499-45b7-b36b-8b011aed4640","Type":"ContainerDied","Data":"6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1"}
Sep 17 17:59:01.584675 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.584617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj" event={"ID":"654d0db9-d499-45b7-b36b-8b011aed4640","Type":"ContainerDied","Data":"0f8cbea59ded75991789494599442e998bdb8d854a44fa1b36540e70f2ef60ad"}
Sep 17 17:59:01.584675 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.584640    2581 scope.go:117] "RemoveContainer" containerID="6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1"
Sep 17 17:59:01.594327 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.594310    2581 scope.go:117] "RemoveContainer" containerID="6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1"
Sep 17 17:59:01.594604 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:59:01.594581    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1\": container with ID starting with 6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1 not found: ID does not exist" containerID="6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1"
Sep 17 17:59:01.594653 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.594612    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1"} err="failed to get container status \"6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1\": rpc error: code = NotFound desc = could not find container \"6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1\": container with ID starting with 6a74a65af1995eecc841686a4b65a5f92a799b1463e0a2ded29c4b17aa6c79b1 not found: ID does not exist"
Sep 17 17:59:01.610496 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.610474    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"]
Sep 17 17:59:01.615406 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:01.615385    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-dcaf3-8fd969546-vzrkj"]
Sep 17 17:59:02.417624 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:02.417590    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:02.512343 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:02.512309    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 17:59:02.898821 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:02.898789    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" path="/var/lib/kubelet/pods/654d0db9-d499-45b7-b36b-8b011aed4640/volumes"
Sep 17 17:59:07.417077 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:07.417037    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:08.424554 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:08.424511    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 17:59:12.417483 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:12.417441    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:12.512777 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:12.512737    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 17:59:14.650015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:14.649978    2581 generic.go:356] "Generic (PLEG): container finished" podID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerID="d708ca9a38f8e5b21b9cdbaadc1777193e017b394b9300c47a47c28cc3967955" exitCode=0
Sep 17 17:59:14.650380 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:14.650045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" event={"ID":"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36","Type":"ContainerDied","Data":"d708ca9a38f8e5b21b9cdbaadc1777193e017b394b9300c47a47c28cc3967955"}
Sep 17 17:59:15.145969 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.145945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:59:15.290934 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.290852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-proxy-tls\") pod \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\" (UID: \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\") "
Sep 17 17:59:15.290934 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.290888    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-openshift-service-ca-bundle\") pod \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\" (UID: \"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36\") "
Sep 17 17:59:15.291242 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.291214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" (UID: "71f6bdcd-2728-4385-bec1-9f4fc1bcdc36"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:15.292924 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.292901    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" (UID: "71f6bdcd-2728-4385-bec1-9f4fc1bcdc36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:15.391959 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.391922    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:59:15.391959 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.391953    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:59:15.656350 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.656263    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"
Sep 17 17:59:15.656350 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.656296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd" event={"ID":"71f6bdcd-2728-4385-bec1-9f4fc1bcdc36","Type":"ContainerDied","Data":"f66d5e652405004dbcb31cbb9434905a6244330547d1feeef35db050d435e191"}
Sep 17 17:59:15.656350 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.656349    2581 scope.go:117] "RemoveContainer" containerID="d708ca9a38f8e5b21b9cdbaadc1777193e017b394b9300c47a47c28cc3967955"
Sep 17 17:59:15.680880 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.680852    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"]
Sep 17 17:59:15.690340 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:15.690313    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1fda1-756f77b956-96xnd"]
Sep 17 17:59:16.898648 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:16.898616    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" path="/var/lib/kubelet/pods/71f6bdcd-2728-4385-bec1-9f4fc1bcdc36/volumes"
Sep 17 17:59:18.424885 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:18.424845    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:59:22.512936 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:22.512891    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 17:59:30.808818 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.808783    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"]
Sep 17 17:59:30.809197 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809147    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container"
Sep 17 17:59:30.809197 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809159    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container"
Sep 17 17:59:30.809197 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809174    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3"
Sep 17 17:59:30.809197 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809180    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3"
Sep 17 17:59:30.809197 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809190    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kube-rbac-proxy"
Sep 17 17:59:30.809197 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809195    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kube-rbac-proxy"
Sep 17 17:59:30.809392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809202    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1"
Sep 17 17:59:30.809392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809208    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1"
Sep 17 17:59:30.809392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809258    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kube-rbac-proxy"
Sep 17 17:59:30.809392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809266    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cba543-f1b7-47ad-b3bc-0d1eac81f333" containerName="kserve-container"
Sep 17 17:59:30.809392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809274    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f6bdcd-2728-4385-bec1-9f4fc1bcdc36" containerName="ensemble-graph-1fda1"
Sep 17 17:59:30.809392 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.809280    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="654d0db9-d499-45b7-b36b-8b011aed4640" containerName="sequence-graph-dcaf3"
Sep 17 17:59:30.813578 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.813559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:30.816275 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.816254    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c397f-serving-cert\""
Sep 17 17:59:30.816275 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.816267    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c397f-kube-rbac-proxy-sar-config\""
Sep 17 17:59:30.822751 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.822727    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"]
Sep 17 17:59:30.926887 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.926848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-openshift-service-ca-bundle\") pod \"sequence-graph-c397f-66d6dcb945-6dwwc\" (UID: \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\") " pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:30.927063 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:30.926919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-proxy-tls\") pod \"sequence-graph-c397f-66d6dcb945-6dwwc\" (UID: \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\") " pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:31.028538 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.028491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-openshift-service-ca-bundle\") pod \"sequence-graph-c397f-66d6dcb945-6dwwc\" (UID: \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\") " pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:31.028748 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.028650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-proxy-tls\") pod \"sequence-graph-c397f-66d6dcb945-6dwwc\" (UID: \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\") " pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:31.029239 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.029206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-openshift-service-ca-bundle\") pod \"sequence-graph-c397f-66d6dcb945-6dwwc\" (UID: \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\") " pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:31.031031 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.031006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-proxy-tls\") pod \"sequence-graph-c397f-66d6dcb945-6dwwc\" (UID: \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\") " pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:31.124057 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.123965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:31.397572 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.397547    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"]
Sep 17 17:59:31.399753 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:59:31.399728    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cd9ea9_793d_48a1_b8fb_cb70ab6edb07.slice/crio-aa28355bc638202f086204bacbc8173030e9825f55022c1b8a653328a09a7cbd WatchSource:0}: Error finding container aa28355bc638202f086204bacbc8173030e9825f55022c1b8a653328a09a7cbd: Status 404 returned error can't find the container with id aa28355bc638202f086204bacbc8173030e9825f55022c1b8a653328a09a7cbd
Sep 17 17:59:31.734734 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.734636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" event={"ID":"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07","Type":"ContainerStarted","Data":"fed2d6a32c77707c349c7a52de4b5aa329eedff7347775cf412f1406ed86a1a4"}
Sep 17 17:59:31.734734 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.734689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" event={"ID":"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07","Type":"ContainerStarted","Data":"aa28355bc638202f086204bacbc8173030e9825f55022c1b8a653328a09a7cbd"}
Sep 17 17:59:31.734914 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.734747    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:31.755150 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:31.755100    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" podStartSLOduration=1.755083515 podStartE2EDuration="1.755083515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:31.754946084 +0000 UTC m=+1486.986976903" watchObservedRunningTime="2026-09-17 17:59:31.755083515 +0000 UTC m=+1486.987114327"
Sep 17 17:59:32.513649 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:32.513621    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 17:59:37.745623 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:37.745593    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:40.867585 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:40.867552    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"]
Sep 17 17:59:40.868117 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:40.867840    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f" containerID="cri-o://fed2d6a32c77707c349c7a52de4b5aa329eedff7347775cf412f1406ed86a1a4" gracePeriod=30
Sep 17 17:59:41.046469 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.046436    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"]
Sep 17 17:59:41.046730 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.046667    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container" containerID="cri-o://483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620" gracePeriod=30
Sep 17 17:59:41.046877 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.046771    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kube-rbac-proxy" containerID="cri-o://3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6" gracePeriod=30
Sep 17 17:59:41.258494 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.258452    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"]
Sep 17 17:59:41.262000 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.261981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.265043 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.265025    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1e4e7-kube-rbac-proxy-sar-config\""
Sep 17 17:59:41.266162 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.266148    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1e4e7-predictor-serving-cert\""
Sep 17 17:59:41.274063 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.274040    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"]
Sep 17 17:59:41.317866 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.317837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.318006 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.317913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2tq\" (UniqueName: \"kubernetes.io/projected/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-api-access-2h2tq\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.318006 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.317936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.419332 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.419249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.419504 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.419334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2tq\" (UniqueName: \"kubernetes.io/projected/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-api-access-2h2tq\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.419504 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.419355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.419504 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:59:41.419465    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-serving-cert: secret "error-404-isvc-1e4e7-predictor-serving-cert" not found
Sep 17 17:59:41.419649 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:59:41.419519    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls podName:7637bb8f-4cf3-4980-8031-8c3862060f3b nodeName:}" failed. No retries permitted until 2026-09-17 17:59:41.919503823 +0000 UTC m=+1497.151534625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls") pod "error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" (UID: "7637bb8f-4cf3-4980-8031-8c3862060f3b") : secret "error-404-isvc-1e4e7-predictor-serving-cert" not found
Sep 17 17:59:41.419988 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.419964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.428781 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.428755    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2tq\" (UniqueName: \"kubernetes.io/projected/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-api-access-2h2tq\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.786365 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.786330    2581 generic.go:356] "Generic (PLEG): container finished" podID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerID="3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6" exitCode=2
Sep 17 17:59:41.786533 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.786388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" event={"ID":"21b643b5-d440-4a78-8c63-f1870f0a20fd","Type":"ContainerDied","Data":"3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6"}
Sep 17 17:59:41.922611 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.922573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:41.924997 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:41.924970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls\") pod \"error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") " pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:42.172435 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.172335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:42.451143 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.451070    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"]
Sep 17 17:59:42.454430 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:59:42.454404    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7637bb8f_4cf3_4980_8031_8c3862060f3b.slice/crio-4310f6d5c7a45f6211233165207662b6f442f333e8aedecc87720500a3e9b707 WatchSource:0}: Error finding container 4310f6d5c7a45f6211233165207662b6f442f333e8aedecc87720500a3e9b707: Status 404 returned error can't find the container with id 4310f6d5c7a45f6211233165207662b6f442f333e8aedecc87720500a3e9b707
Sep 17 17:59:42.745694 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.745650    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:42.793150 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.793115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" event={"ID":"7637bb8f-4cf3-4980-8031-8c3862060f3b","Type":"ContainerStarted","Data":"40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d"}
Sep 17 17:59:42.793439 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.793189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:42.793439 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.793216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" event={"ID":"7637bb8f-4cf3-4980-8031-8c3862060f3b","Type":"ContainerStarted","Data":"530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f"}
Sep 17 17:59:42.793439 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.793341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:42.793439 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.793381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" event={"ID":"7637bb8f-4cf3-4980-8031-8c3862060f3b","Type":"ContainerStarted","Data":"4310f6d5c7a45f6211233165207662b6f442f333e8aedecc87720500a3e9b707"}
Sep 17 17:59:42.794543 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.794518    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 17:59:42.816259 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:42.816212    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podStartSLOduration=1.8161947600000001 podStartE2EDuration="1.81619476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:42.815098352 +0000 UTC m=+1498.047129174" watchObservedRunningTime="2026-09-17 17:59:42.81619476 +0000 UTC m=+1498.048225580"
Sep 17 17:59:43.420127 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:43.420085    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 17:59:43.798323 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:43.798277    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 17:59:44.196175 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.196151    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:59:44.243532 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.243495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21b643b5-d440-4a78-8c63-f1870f0a20fd-proxy-tls\") pod \"21b643b5-d440-4a78-8c63-f1870f0a20fd\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") "
Sep 17 17:59:44.243697 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.243581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvf9j\" (UniqueName: \"kubernetes.io/projected/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-api-access-dvf9j\") pod \"21b643b5-d440-4a78-8c63-f1870f0a20fd\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") "
Sep 17 17:59:44.243697 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.243667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-rbac-proxy-sar-config\") pod \"21b643b5-d440-4a78-8c63-f1870f0a20fd\" (UID: \"21b643b5-d440-4a78-8c63-f1870f0a20fd\") "
Sep 17 17:59:44.244073 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.244047    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21b643b5-d440-4a78-8c63-f1870f0a20fd" (UID: "21b643b5-d440-4a78-8c63-f1870f0a20fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:44.245677 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.245652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-api-access-dvf9j" (OuterVolumeSpecName: "kube-api-access-dvf9j") pod "21b643b5-d440-4a78-8c63-f1870f0a20fd" (UID: "21b643b5-d440-4a78-8c63-f1870f0a20fd"). InnerVolumeSpecName "kube-api-access-dvf9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:44.245862 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.245841    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b643b5-d440-4a78-8c63-f1870f0a20fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21b643b5-d440-4a78-8c63-f1870f0a20fd" (UID: "21b643b5-d440-4a78-8c63-f1870f0a20fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:44.344743 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.344643    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:59:44.344743 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.344674    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21b643b5-d440-4a78-8c63-f1870f0a20fd-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:59:44.344743 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.344684    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvf9j\" (UniqueName: \"kubernetes.io/projected/21b643b5-d440-4a78-8c63-f1870f0a20fd-kube-api-access-dvf9j\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 17:59:44.804278 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.804238    2581 generic.go:356] "Generic (PLEG): container finished" podID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerID="483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620" exitCode=0
Sep 17 17:59:44.804760 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.804292    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"
Sep 17 17:59:44.804760 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.804327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" event={"ID":"21b643b5-d440-4a78-8c63-f1870f0a20fd","Type":"ContainerDied","Data":"483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620"}
Sep 17 17:59:44.804760 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.804401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5" event={"ID":"21b643b5-d440-4a78-8c63-f1870f0a20fd","Type":"ContainerDied","Data":"f53e226e4f5238c8fed574d076702633ac981c2cf720125ebd6c25079feccdd6"}
Sep 17 17:59:44.804760 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.804445    2581 scope.go:117] "RemoveContainer" containerID="3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6"
Sep 17 17:59:44.813644 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.813628    2581 scope.go:117] "RemoveContainer" containerID="483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620"
Sep 17 17:59:44.820856 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.820838    2581 scope.go:117] "RemoveContainer" containerID="3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6"
Sep 17 17:59:44.821137 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:59:44.821120    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6\": container with ID starting with 3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6 not found: ID does not exist" containerID="3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6"
Sep 17 17:59:44.821189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.821146    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6"} err="failed to get container status \"3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6\": rpc error: code = NotFound desc = could not find container \"3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6\": container with ID starting with 3e2fed004e9be850d2232c66cc47d5fdf0e9344ef29f2f5e97b08b3cc98f07f6 not found: ID does not exist"
Sep 17 17:59:44.821189 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.821169    2581 scope.go:117] "RemoveContainer" containerID="483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620"
Sep 17 17:59:44.821406 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:59:44.821386    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620\": container with ID starting with 483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620 not found: ID does not exist" containerID="483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620"
Sep 17 17:59:44.821444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.821411    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620"} err="failed to get container status \"483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620\": rpc error: code = NotFound desc = could not find container \"483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620\": container with ID starting with 483f53f6ac2425b5f68bc76190d4a355df10a4be6b4e290c0de50280071ef620 not found: ID does not exist"
Sep 17 17:59:44.829795 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.829772    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"]
Sep 17 17:59:44.839244 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.839210    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c397f-predictor-699fb4686-ws4j5"]
Sep 17 17:59:44.898612 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:44.898584    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" path="/var/lib/kubelet/pods/21b643b5-d440-4a78-8c63-f1870f0a20fd/volumes"
Sep 17 17:59:46.876990 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:46.876958    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:59:46.878236 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:46.878215    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 17:59:47.744881 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:47.744834    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:48.803179 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:48.803146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 17:59:48.803649 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:48.803621    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 17:59:52.744101 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:52.744062    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:52.744502 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:52.744178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 17:59:54.819160 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.819127    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"]
Sep 17 17:59:54.819561 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.819513    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container"
Sep 17 17:59:54.819561 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.819524    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container"
Sep 17 17:59:54.819561 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.819532    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kube-rbac-proxy"
Sep 17 17:59:54.819561 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.819537    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kube-rbac-proxy"
Sep 17 17:59:54.819727 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.819598    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kube-rbac-proxy"
Sep 17 17:59:54.819727 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.819609    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b643b5-d440-4a78-8c63-f1870f0a20fd" containerName="kserve-container"
Sep 17 17:59:54.824371 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.824349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:54.827912 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.827891    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-1175d-serving-cert\""
Sep 17 17:59:54.828015 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.827939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-1175d-kube-rbac-proxy-sar-config\""
Sep 17 17:59:54.836213 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.836188    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"]
Sep 17 17:59:54.938890 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.938858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls\") pod \"ensemble-graph-1175d-68b6c76bfd-vkdb6\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") " pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:54.939065 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:54.938985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-openshift-service-ca-bundle\") pod \"ensemble-graph-1175d-68b6c76bfd-vkdb6\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") " pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:55.040534 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:55.040491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-openshift-service-ca-bundle\") pod \"ensemble-graph-1175d-68b6c76bfd-vkdb6\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") " pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:55.040753 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:55.040614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls\") pod \"ensemble-graph-1175d-68b6c76bfd-vkdb6\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") " pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:55.040840 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:59:55.040772    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-1175d-serving-cert: secret "ensemble-graph-1175d-serving-cert" not found
Sep 17 17:59:55.040901 ip-10-0-133-107 kubenswrapper[2581]: E0917 17:59:55.040849    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls podName:de3fbad8-f28a-4f54-80ef-8dde100dcbb0 nodeName:}" failed. No retries permitted until 2026-09-17 17:59:55.540830891 +0000 UTC m=+1510.772861687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls") pod "ensemble-graph-1175d-68b6c76bfd-vkdb6" (UID: "de3fbad8-f28a-4f54-80ef-8dde100dcbb0") : secret "ensemble-graph-1175d-serving-cert" not found
Sep 17 17:59:55.041246 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:55.041221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-openshift-service-ca-bundle\") pod \"ensemble-graph-1175d-68b6c76bfd-vkdb6\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") " pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:55.544416 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:55.544373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls\") pod \"ensemble-graph-1175d-68b6c76bfd-vkdb6\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") " pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:55.546791 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:55.546773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls\") pod \"ensemble-graph-1175d-68b6c76bfd-vkdb6\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") " pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:55.734720 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:55.734676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:56.006826 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:56.006629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"]
Sep 17 17:59:56.009722 ip-10-0-133-107 kubenswrapper[2581]: W0917 17:59:56.009680    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3fbad8_f28a_4f54_80ef_8dde100dcbb0.slice/crio-c3cbb07d595f9c1e7c8bbe1465d3847c11979387aed782e7d5e3e778c73d63c7 WatchSource:0}: Error finding container c3cbb07d595f9c1e7c8bbe1465d3847c11979387aed782e7d5e3e778c73d63c7: Status 404 returned error can't find the container with id c3cbb07d595f9c1e7c8bbe1465d3847c11979387aed782e7d5e3e778c73d63c7
Sep 17 17:59:56.868258 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:56.868225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" event={"ID":"de3fbad8-f28a-4f54-80ef-8dde100dcbb0","Type":"ContainerStarted","Data":"f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137"}
Sep 17 17:59:56.868444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:56.868276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" event={"ID":"de3fbad8-f28a-4f54-80ef-8dde100dcbb0","Type":"ContainerStarted","Data":"c3cbb07d595f9c1e7c8bbe1465d3847c11979387aed782e7d5e3e778c73d63c7"}
Sep 17 17:59:56.868444 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:56.868332    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 17:59:56.890344 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:56.890303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" podStartSLOduration=2.890289169 podStartE2EDuration="2.890289169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:56.888402015 +0000 UTC m=+1512.120432833" watchObservedRunningTime="2026-09-17 17:59:56.890289169 +0000 UTC m=+1512.122319988"
Sep 17 17:59:57.745069 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:57.744985    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:59:58.803784 ip-10-0-133-107 kubenswrapper[2581]: I0917 17:59:58.803745    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:00:02.745324 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:02.745279    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:00:02.877993 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:02.877959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 18:00:07.744636 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:07.744598    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:00:08.803931 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:08.803889    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:00:10.938674 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:10.938629    2581 generic.go:356] "Generic (PLEG): container finished" podID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerID="fed2d6a32c77707c349c7a52de4b5aa329eedff7347775cf412f1406ed86a1a4" exitCode=0
Sep 17 18:00:10.939122 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:10.938719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" event={"ID":"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07","Type":"ContainerDied","Data":"fed2d6a32c77707c349c7a52de4b5aa329eedff7347775cf412f1406ed86a1a4"}
Sep 17 18:00:11.510119 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.510096    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 18:00:11.583024 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.582997    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-openshift-service-ca-bundle\") pod \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\" (UID: \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\") "
Sep 17 18:00:11.583171 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.583061    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-proxy-tls\") pod \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\" (UID: \"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07\") "
Sep 17 18:00:11.583374 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.583352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" (UID: "92cd9ea9-793d-48a1-b8fb-cb70ab6edb07"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:11.585046 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.585024    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" (UID: "92cd9ea9-793d-48a1-b8fb-cb70ab6edb07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:11.684371 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.684286    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.684371 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.684324    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:00:11.945172 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.945095    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"
Sep 17 18:00:11.945542 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.945099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc" event={"ID":"92cd9ea9-793d-48a1-b8fb-cb70ab6edb07","Type":"ContainerDied","Data":"aa28355bc638202f086204bacbc8173030e9825f55022c1b8a653328a09a7cbd"}
Sep 17 18:00:11.945542 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.945223    2581 scope.go:117] "RemoveContainer" containerID="fed2d6a32c77707c349c7a52de4b5aa329eedff7347775cf412f1406ed86a1a4"
Sep 17 18:00:11.981341 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.981317    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"]
Sep 17 18:00:11.996219 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:11.996194    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c397f-66d6dcb945-6dwwc"]
Sep 17 18:00:12.898654 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:12.898621    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" path="/var/lib/kubelet/pods/92cd9ea9-793d-48a1-b8fb-cb70ab6edb07/volumes"
Sep 17 18:00:18.804641 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:18.804601    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:00:28.804850 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:28.804819    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 18:00:41.118806 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.118772    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"]
Sep 17 18:00:41.119180 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.119136    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f"
Sep 17 18:00:41.119180 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.119146    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f"
Sep 17 18:00:41.119264 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.119211    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cd9ea9-793d-48a1-b8fb-cb70ab6edb07" containerName="sequence-graph-c397f"
Sep 17 18:00:41.122280 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.122263    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:41.124870 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.124849    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-1e4e7-kube-rbac-proxy-sar-config\""
Sep 17 18:00:41.124987 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.124935    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-1e4e7-serving-cert\""
Sep 17 18:00:41.131993 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.131972    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"]
Sep 17 18:00:41.228147 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.228102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ce48c7-65f2-4590-917b-423b51760da6-openshift-service-ca-bundle\") pod \"sequence-graph-1e4e7-6f5ffcc466-vpjqm\" (UID: \"71ce48c7-65f2-4590-917b-423b51760da6\") " pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:41.228354 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.228162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ce48c7-65f2-4590-917b-423b51760da6-proxy-tls\") pod \"sequence-graph-1e4e7-6f5ffcc466-vpjqm\" (UID: \"71ce48c7-65f2-4590-917b-423b51760da6\") " pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:41.329171 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.329138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ce48c7-65f2-4590-917b-423b51760da6-proxy-tls\") pod \"sequence-graph-1e4e7-6f5ffcc466-vpjqm\" (UID: \"71ce48c7-65f2-4590-917b-423b51760da6\") " pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:41.329356 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.329224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ce48c7-65f2-4590-917b-423b51760da6-openshift-service-ca-bundle\") pod \"sequence-graph-1e4e7-6f5ffcc466-vpjqm\" (UID: \"71ce48c7-65f2-4590-917b-423b51760da6\") " pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:41.329897 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.329874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ce48c7-65f2-4590-917b-423b51760da6-openshift-service-ca-bundle\") pod \"sequence-graph-1e4e7-6f5ffcc466-vpjqm\" (UID: \"71ce48c7-65f2-4590-917b-423b51760da6\") " pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:41.331622 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.331603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ce48c7-65f2-4590-917b-423b51760da6-proxy-tls\") pod \"sequence-graph-1e4e7-6f5ffcc466-vpjqm\" (UID: \"71ce48c7-65f2-4590-917b-423b51760da6\") " pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:41.432917 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.432841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:41.707911 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:41.707827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"]
Sep 17 18:00:41.711233 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:00:41.711205    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ce48c7_65f2_4590_917b_423b51760da6.slice/crio-16367af1ca4202414653cbda636fe51113d9d6a91bb94ce968cec1299e259cae WatchSource:0}: Error finding container 16367af1ca4202414653cbda636fe51113d9d6a91bb94ce968cec1299e259cae: Status 404 returned error can't find the container with id 16367af1ca4202414653cbda636fe51113d9d6a91bb94ce968cec1299e259cae
Sep 17 18:00:42.085725 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:42.085675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" event={"ID":"71ce48c7-65f2-4590-917b-423b51760da6","Type":"ContainerStarted","Data":"55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d"}
Sep 17 18:00:42.085959 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:42.085747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" event={"ID":"71ce48c7-65f2-4590-917b-423b51760da6","Type":"ContainerStarted","Data":"16367af1ca4202414653cbda636fe51113d9d6a91bb94ce968cec1299e259cae"}
Sep 17 18:00:42.085959 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:42.085820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:00:42.111053 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:42.111006    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" podStartSLOduration=1.110989832 podStartE2EDuration="1.110989832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:42.10959145 +0000 UTC m=+1557.341622270" watchObservedRunningTime="2026-09-17 18:00:42.110989832 +0000 UTC m=+1557.343020670"
Sep 17 18:00:48.096321 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:00:48.096292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:04:46.908235 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:04:46.908209    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:04:46.909889 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:04:46.909863    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:08:09.455397 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.455325    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"]
Sep 17 18:08:09.457911 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.455531    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d" containerID="cri-o://f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137" gracePeriod=30
Sep 17 18:08:09.792802 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.792772    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"]
Sep 17 18:08:09.793003 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.792983    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container" containerID="cri-o://4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137" gracePeriod=30
Sep 17 18:08:09.793737 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.793481    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kube-rbac-proxy" containerID="cri-o://cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b" gracePeriod=30
Sep 17 18:08:09.924364 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.924331    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"]
Sep 17 18:08:09.927694 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.927679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:09.936679 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.936657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9819a-predictor-serving-cert\""
Sep 17 18:08:09.936803 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.936689    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9819a-kube-rbac-proxy-sar-config\""
Sep 17 18:08:09.946986 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:09.946959    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"]
Sep 17 18:08:10.015660 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.015630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.015786 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.015675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.015833 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.015800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxvb6\" (UniqueName: \"kubernetes.io/projected/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-api-access-fxvb6\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.116989 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.116934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxvb6\" (UniqueName: \"kubernetes.io/projected/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-api-access-fxvb6\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.116989 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.116987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.117151 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.117013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.117151 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:08:10.117131    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9819a-predictor-serving-cert: secret "error-404-isvc-9819a-predictor-serving-cert" not found
Sep 17 18:08:10.117245 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:08:10.117203    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls podName:07a60588-4a99-49d3-9ac1-adae9b1f8c69 nodeName:}" failed. No retries permitted until 2026-09-17 18:08:10.617183205 +0000 UTC m=+2005.849214005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls") pod "error-404-isvc-9819a-predictor-75b9f568db-gpdmf" (UID: "07a60588-4a99-49d3-9ac1-adae9b1f8c69") : secret "error-404-isvc-9819a-predictor-serving-cert" not found
Sep 17 18:08:10.117600 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.117584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.144522 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.144503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxvb6\" (UniqueName: \"kubernetes.io/projected/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-api-access-fxvb6\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.166505 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.166481    2581 generic.go:356] "Generic (PLEG): container finished" podID="53d6cea0-bed8-4b87-a22e-a770be732246" containerID="cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b" exitCode=2
Sep 17 18:08:10.166602 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.166510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" event={"ID":"53d6cea0-bed8-4b87-a22e-a770be732246","Type":"ContainerDied","Data":"cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b"}
Sep 17 18:08:10.621643 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.621608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.624087 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.624068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls\") pod \"error-404-isvc-9819a-predictor-75b9f568db-gpdmf\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") " pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:10.836977 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:10.836947    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:11.127366 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:11.127341    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"]
Sep 17 18:08:11.129970 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:08:11.129939    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a60588_4a99_49d3_9ac1_adae9b1f8c69.slice/crio-8592dc69c2de48f3388ffc077cc45e8d9860643452e50926cac105ed8f9fbf8d WatchSource:0}: Error finding container 8592dc69c2de48f3388ffc077cc45e8d9860643452e50926cac105ed8f9fbf8d: Status 404 returned error can't find the container with id 8592dc69c2de48f3388ffc077cc45e8d9860643452e50926cac105ed8f9fbf8d
Sep 17 18:08:11.172887 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:11.172862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" event={"ID":"07a60588-4a99-49d3-9ac1-adae9b1f8c69","Type":"ContainerStarted","Data":"8592dc69c2de48f3388ffc077cc45e8d9860643452e50926cac105ed8f9fbf8d"}
Sep 17 18:08:12.179651 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.179614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" event={"ID":"07a60588-4a99-49d3-9ac1-adae9b1f8c69","Type":"ContainerStarted","Data":"6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822"}
Sep 17 18:08:12.180060 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.179673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" event={"ID":"07a60588-4a99-49d3-9ac1-adae9b1f8c69","Type":"ContainerStarted","Data":"41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a"}
Sep 17 18:08:12.180060 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.179738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:12.201895 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.201844    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podStartSLOduration=3.201828376 podStartE2EDuration="3.201828376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:12.200401647 +0000 UTC m=+2007.432432489" watchObservedRunningTime="2026-09-17 18:08:12.201828376 +0000 UTC m=+2007.433859194"
Sep 17 18:08:12.507227 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.507195    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 18:08:12.512402 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.512377    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:08:12.653624 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.653601    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 18:08:12.739402 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.739334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hl9b\" (UniqueName: \"kubernetes.io/projected/53d6cea0-bed8-4b87-a22e-a770be732246-kube-api-access-7hl9b\") pod \"53d6cea0-bed8-4b87-a22e-a770be732246\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") "
Sep 17 18:08:12.739402 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.739370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d6cea0-bed8-4b87-a22e-a770be732246-proxy-tls\") pod \"53d6cea0-bed8-4b87-a22e-a770be732246\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") "
Sep 17 18:08:12.739566 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.739430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d6cea0-bed8-4b87-a22e-a770be732246-kube-rbac-proxy-sar-config\") pod \"53d6cea0-bed8-4b87-a22e-a770be732246\" (UID: \"53d6cea0-bed8-4b87-a22e-a770be732246\") "
Sep 17 18:08:12.739897 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.739867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d6cea0-bed8-4b87-a22e-a770be732246-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53d6cea0-bed8-4b87-a22e-a770be732246" (UID: "53d6cea0-bed8-4b87-a22e-a770be732246"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:12.741437 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.741402    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d6cea0-bed8-4b87-a22e-a770be732246-kube-api-access-7hl9b" (OuterVolumeSpecName: "kube-api-access-7hl9b") pod "53d6cea0-bed8-4b87-a22e-a770be732246" (UID: "53d6cea0-bed8-4b87-a22e-a770be732246"). InnerVolumeSpecName "kube-api-access-7hl9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:12.741533 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.741504    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53d6cea0-bed8-4b87-a22e-a770be732246-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53d6cea0-bed8-4b87-a22e-a770be732246" (UID: "53d6cea0-bed8-4b87-a22e-a770be732246"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:12.840637 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.840614    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53d6cea0-bed8-4b87-a22e-a770be732246-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:08:12.840637 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.840635    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hl9b\" (UniqueName: \"kubernetes.io/projected/53d6cea0-bed8-4b87-a22e-a770be732246-kube-api-access-7hl9b\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:08:12.840798 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.840645    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53d6cea0-bed8-4b87-a22e-a770be732246-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:08:12.876986 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:12.876961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:13.185429 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.185399    2581 generic.go:356] "Generic (PLEG): container finished" podID="53d6cea0-bed8-4b87-a22e-a770be732246" containerID="4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137" exitCode=0
Sep 17 18:08:13.185822 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.185448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"
Sep 17 18:08:13.185822 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.185479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" event={"ID":"53d6cea0-bed8-4b87-a22e-a770be732246","Type":"ContainerDied","Data":"4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137"}
Sep 17 18:08:13.185822 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.185533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r" event={"ID":"53d6cea0-bed8-4b87-a22e-a770be732246","Type":"ContainerDied","Data":"16d9a1f922368d80b237d0ad5eff263143c415e3ee61d4133c75a15994855eee"}
Sep 17 18:08:13.185822 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.185558    2581 scope.go:117] "RemoveContainer" containerID="cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b"
Sep 17 18:08:13.186041 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.185970    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:13.187731 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.187683    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:08:13.194294 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.194275    2581 scope.go:117] "RemoveContainer" containerID="4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137"
Sep 17 18:08:13.201777 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.201753    2581 scope.go:117] "RemoveContainer" containerID="cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b"
Sep 17 18:08:13.202032 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:08:13.202013    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b\": container with ID starting with cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b not found: ID does not exist" containerID="cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b"
Sep 17 18:08:13.202111 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.202038    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b"} err="failed to get container status \"cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b\": rpc error: code = NotFound desc = could not find container \"cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b\": container with ID starting with cb524ee31b55e924c7a46e3ec3223c806cb332116d33505434d5835e4875837b not found: ID does not exist"
Sep 17 18:08:13.202111 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.202055    2581 scope.go:117] "RemoveContainer" containerID="4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137"
Sep 17 18:08:13.202320 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:08:13.202304    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137\": container with ID starting with 4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137 not found: ID does not exist" containerID="4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137"
Sep 17 18:08:13.202366 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.202342    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137"} err="failed to get container status \"4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137\": rpc error: code = NotFound desc = could not find container \"4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137\": container with ID starting with 4c7ae36b5da67592c2f81b2245fdd9a3517620a626821ea1d19ce17226fd7137 not found: ID does not exist"
Sep 17 18:08:13.220222 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.220198    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"]
Sep 17 18:08:13.222744 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:13.222724    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1175d-predictor-6784b69858-d8x9r"]
Sep 17 18:08:14.190895 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:14.190858    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:08:14.898438 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:14.898408    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" path="/var/lib/kubelet/pods/53d6cea0-bed8-4b87-a22e-a770be732246/volumes"
Sep 17 18:08:17.876917 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:17.876874    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:19.194835 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:19.194809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:19.195232 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:19.195202    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:08:22.877130 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:22.877083    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:22.877520 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:22.877196    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 18:08:27.877373 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:27.877327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:29.196075 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:29.196040    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:08:32.876561 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:32.876515    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:37.877275 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:37.877232    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:39.195246 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:39.195207    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:08:39.601556 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:39.601530    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 18:08:39.752127 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:39.752103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls\") pod \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") "
Sep 17 18:08:39.752259 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:39.752204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-openshift-service-ca-bundle\") pod \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\" (UID: \"de3fbad8-f28a-4f54-80ef-8dde100dcbb0\") "
Sep 17 18:08:39.752553 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:39.752528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "de3fbad8-f28a-4f54-80ef-8dde100dcbb0" (UID: "de3fbad8-f28a-4f54-80ef-8dde100dcbb0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:39.754184 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:39.754158    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de3fbad8-f28a-4f54-80ef-8dde100dcbb0" (UID: "de3fbad8-f28a-4f54-80ef-8dde100dcbb0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:39.853552 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:39.853528    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:08:39.853552 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:39.853551    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3fbad8-f28a-4f54-80ef-8dde100dcbb0-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:08:40.322908 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.322875    2581 generic.go:356] "Generic (PLEG): container finished" podID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerID="f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137" exitCode=0
Sep 17 18:08:40.323478 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.322924    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"
Sep 17 18:08:40.323478 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.322957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" event={"ID":"de3fbad8-f28a-4f54-80ef-8dde100dcbb0","Type":"ContainerDied","Data":"f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137"}
Sep 17 18:08:40.323478 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.323001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6" event={"ID":"de3fbad8-f28a-4f54-80ef-8dde100dcbb0","Type":"ContainerDied","Data":"c3cbb07d595f9c1e7c8bbe1465d3847c11979387aed782e7d5e3e778c73d63c7"}
Sep 17 18:08:40.323478 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.323022    2581 scope.go:117] "RemoveContainer" containerID="f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137"
Sep 17 18:08:40.334144 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.334123    2581 scope.go:117] "RemoveContainer" containerID="f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137"
Sep 17 18:08:40.334411 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:08:40.334390    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137\": container with ID starting with f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137 not found: ID does not exist" containerID="f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137"
Sep 17 18:08:40.334493 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.334420    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137"} err="failed to get container status \"f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137\": rpc error: code = NotFound desc = could not find container \"f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137\": container with ID starting with f949ac5c2bba88194103e024970e78f85ffa2016dac6977135a4561a35142137 not found: ID does not exist"
Sep 17 18:08:40.356403 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.356381    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"]
Sep 17 18:08:40.364005 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.363972    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1175d-68b6c76bfd-vkdb6"]
Sep 17 18:08:40.898962 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:40.898933    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" path="/var/lib/kubelet/pods/de3fbad8-f28a-4f54-80ef-8dde100dcbb0/volumes"
Sep 17 18:08:49.196182 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:49.196144    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:08:55.797368 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:55.797332    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"]
Sep 17 18:08:55.797797 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:55.797525    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7" containerID="cri-o://55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d" gracePeriod=30
Sep 17 18:08:56.071007 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.070927    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"]
Sep 17 18:08:56.071224 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.071165    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container" containerID="cri-o://530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f" gracePeriod=30
Sep 17 18:08:56.071312 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.071196    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kube-rbac-proxy" containerID="cri-o://40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d" gracePeriod=30
Sep 17 18:08:56.277739 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.277683    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"]
Sep 17 18:08:56.278091 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278077    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container"
Sep 17 18:08:56.278137 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278094    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container"
Sep 17 18:08:56.278137 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kube-rbac-proxy"
Sep 17 18:08:56.278137 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278120    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kube-rbac-proxy"
Sep 17 18:08:56.278137 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278127    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d"
Sep 17 18:08:56.278137 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278132    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d"
Sep 17 18:08:56.278287 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278189    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kserve-container"
Sep 17 18:08:56.278287 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278196    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="53d6cea0-bed8-4b87-a22e-a770be732246" containerName="kube-rbac-proxy"
Sep 17 18:08:56.278287 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.278203    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3fbad8-f28a-4f54-80ef-8dde100dcbb0" containerName="ensemble-graph-1175d"
Sep 17 18:08:56.281250 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.281229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.284416 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.284392    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5ec4c-predictor-serving-cert\""
Sep 17 18:08:56.284416 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.284407    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5ec4c-kube-rbac-proxy-sar-config\""
Sep 17 18:08:56.297905 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.297881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"]
Sep 17 18:08:56.378318 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.378241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.378318 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.378306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85aed06b-af8a-410d-9c0d-a76d1722e3fd-proxy-tls\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.378484 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.378361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fznh\" (UniqueName: \"kubernetes.io/projected/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-api-access-4fznh\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.400657 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.400629    2581 generic.go:356] "Generic (PLEG): container finished" podID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerID="40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d" exitCode=2
Sep 17 18:08:56.400809 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.400712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" event={"ID":"7637bb8f-4cf3-4980-8031-8c3862060f3b","Type":"ContainerDied","Data":"40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d"}
Sep 17 18:08:56.479426 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.479395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85aed06b-af8a-410d-9c0d-a76d1722e3fd-proxy-tls\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.479587 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.479433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fznh\" (UniqueName: \"kubernetes.io/projected/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-api-access-4fznh\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.479587 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.479480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.480147 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.480123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.481991 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.481967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85aed06b-af8a-410d-9c0d-a76d1722e3fd-proxy-tls\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.496524 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.496495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fznh\" (UniqueName: \"kubernetes.io/projected/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-api-access-4fznh\") pod \"error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") " pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.591551 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.591528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:56.867327 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:56.867304    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"]
Sep 17 18:08:56.869105 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:08:56.869078    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85aed06b_af8a_410d_9c0d_a76d1722e3fd.slice/crio-c7798913132220b29cf6546e9c2b52b7aba740b9d64f90f515b0b27dbea98f88 WatchSource:0}: Error finding container c7798913132220b29cf6546e9c2b52b7aba740b9d64f90f515b0b27dbea98f88: Status 404 returned error can't find the container with id c7798913132220b29cf6546e9c2b52b7aba740b9d64f90f515b0b27dbea98f88
Sep 17 18:08:57.406893 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:57.406852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" event={"ID":"85aed06b-af8a-410d-9c0d-a76d1722e3fd","Type":"ContainerStarted","Data":"3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859"}
Sep 17 18:08:57.407080 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:57.406922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" event={"ID":"85aed06b-af8a-410d-9c0d-a76d1722e3fd","Type":"ContainerStarted","Data":"93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95"}
Sep 17 18:08:57.407080 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:57.406955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" event={"ID":"85aed06b-af8a-410d-9c0d-a76d1722e3fd","Type":"ContainerStarted","Data":"c7798913132220b29cf6546e9c2b52b7aba740b9d64f90f515b0b27dbea98f88"}
Sep 17 18:08:57.407187 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:57.407115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:57.407187 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:57.407173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:08:57.408335 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:57.408311    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:08:57.427790 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:57.427748    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podStartSLOduration=1.427734295 podStartE2EDuration="1.427734295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:57.42588389 +0000 UTC m=+2052.657914722" watchObservedRunningTime="2026-09-17 18:08:57.427734295 +0000 UTC m=+2052.659765113"
Sep 17 18:08:58.095339 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:58.095295    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:08:58.411285 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:58.411187    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:08:58.798552 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:58.798512    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 18:08:58.803981 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:58.803946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:08:58.918653 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:58.918627    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 18:08:58.999995 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:58.999970    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h2tq\" (UniqueName: \"kubernetes.io/projected/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-api-access-2h2tq\") pod \"7637bb8f-4cf3-4980-8031-8c3862060f3b\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") "
Sep 17 18:08:59.000112 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.000015    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls\") pod \"7637bb8f-4cf3-4980-8031-8c3862060f3b\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") "
Sep 17 18:08:59.000112 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.000049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-rbac-proxy-sar-config\") pod \"7637bb8f-4cf3-4980-8031-8c3862060f3b\" (UID: \"7637bb8f-4cf3-4980-8031-8c3862060f3b\") "
Sep 17 18:08:59.000440 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.000414    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7637bb8f-4cf3-4980-8031-8c3862060f3b" (UID: "7637bb8f-4cf3-4980-8031-8c3862060f3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:59.002008 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.001984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-api-access-2h2tq" (OuterVolumeSpecName: "kube-api-access-2h2tq") pod "7637bb8f-4cf3-4980-8031-8c3862060f3b" (UID: "7637bb8f-4cf3-4980-8031-8c3862060f3b"). InnerVolumeSpecName "kube-api-access-2h2tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:59.002126 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.002107    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7637bb8f-4cf3-4980-8031-8c3862060f3b" (UID: "7637bb8f-4cf3-4980-8031-8c3862060f3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:59.100749 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.100725    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:08:59.100749 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.100746    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2tq\" (UniqueName: \"kubernetes.io/projected/7637bb8f-4cf3-4980-8031-8c3862060f3b-kube-api-access-2h2tq\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:08:59.101074 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.100759    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7637bb8f-4cf3-4980-8031-8c3862060f3b-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:08:59.195870 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.195843    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:08:59.417200 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.417118    2581 generic.go:356] "Generic (PLEG): container finished" podID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerID="530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f" exitCode=0
Sep 17 18:08:59.417200 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.417168    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"
Sep 17 18:08:59.417370 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.417196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" event={"ID":"7637bb8f-4cf3-4980-8031-8c3862060f3b","Type":"ContainerDied","Data":"530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f"}
Sep 17 18:08:59.417370 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.417256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn" event={"ID":"7637bb8f-4cf3-4980-8031-8c3862060f3b","Type":"ContainerDied","Data":"4310f6d5c7a45f6211233165207662b6f442f333e8aedecc87720500a3e9b707"}
Sep 17 18:08:59.417370 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.417286    2581 scope.go:117] "RemoveContainer" containerID="40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d"
Sep 17 18:08:59.426618 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.426600    2581 scope.go:117] "RemoveContainer" containerID="530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f"
Sep 17 18:08:59.434183 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.434162    2581 scope.go:117] "RemoveContainer" containerID="40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d"
Sep 17 18:08:59.434469 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:08:59.434451    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d\": container with ID starting with 40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d not found: ID does not exist" containerID="40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d"
Sep 17 18:08:59.434575 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.434477    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d"} err="failed to get container status \"40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d\": rpc error: code = NotFound desc = could not find container \"40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d\": container with ID starting with 40f187c9cf570d879d4f2ed8d21e22955358a74ebc830b2a27119555d4fac10d not found: ID does not exist"
Sep 17 18:08:59.434575 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.434492    2581 scope.go:117] "RemoveContainer" containerID="530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f"
Sep 17 18:08:59.434835 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:08:59.434801    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f\": container with ID starting with 530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f not found: ID does not exist" containerID="530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f"
Sep 17 18:08:59.434937 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.434837    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f"} err="failed to get container status \"530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f\": rpc error: code = NotFound desc = could not find container \"530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f\": container with ID starting with 530371498abb67056aac0d8a2411a86a2ceb0ab416673070fa8328135258bf0f not found: ID does not exist"
Sep 17 18:08:59.446151 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.446123    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"]
Sep 17 18:08:59.451168 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:08:59.451147    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e4e7-predictor-86f4b8b95d-glqzn"]
Sep 17 18:09:00.899677 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:00.899633    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" path="/var/lib/kubelet/pods/7637bb8f-4cf3-4980-8031-8c3862060f3b/volumes"
Sep 17 18:09:03.094611 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:03.094568    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:03.416454 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:03.416379    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:09:03.416816 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:03.416791    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:09:08.095291 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:08.095252    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:08.095690 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:08.095358    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:09:09.716395 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.716355    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"]
Sep 17 18:09:09.716929 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.716910    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container"
Sep 17 18:09:09.717019 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.716931    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container"
Sep 17 18:09:09.717019 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.716955    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kube-rbac-proxy"
Sep 17 18:09:09.717019 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.716964    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kube-rbac-proxy"
Sep 17 18:09:09.717173 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.717062    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kserve-container"
Sep 17 18:09:09.717173 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.717077    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7637bb8f-4cf3-4980-8031-8c3862060f3b" containerName="kube-rbac-proxy"
Sep 17 18:09:09.719978 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.719956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:09.723506 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.723486    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9819a-serving-cert\""
Sep 17 18:09:09.723581 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.723559    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9819a-kube-rbac-proxy-sar-config\""
Sep 17 18:09:09.730918 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.730895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"]
Sep 17 18:09:09.778875 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.778854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-openshift-service-ca-bundle\") pod \"splitter-graph-9819a-5b6b565754-mn9bp\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") " pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:09.778991 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.778921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls\") pod \"splitter-graph-9819a-5b6b565754-mn9bp\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") " pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:09.880238 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.880207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls\") pod \"splitter-graph-9819a-5b6b565754-mn9bp\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") " pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:09.880388 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.880265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-openshift-service-ca-bundle\") pod \"splitter-graph-9819a-5b6b565754-mn9bp\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") " pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:09.880437 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:09.880381    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-9819a-serving-cert: secret "splitter-graph-9819a-serving-cert" not found
Sep 17 18:09:09.880475 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:09.880444    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls podName:91ba2c39-06ab-4248-8a2c-9f4e1aea920a nodeName:}" failed. No retries permitted until 2026-09-17 18:09:10.380428524 +0000 UTC m=+2065.612459321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls") pod "splitter-graph-9819a-5b6b565754-mn9bp" (UID: "91ba2c39-06ab-4248-8a2c-9f4e1aea920a") : secret "splitter-graph-9819a-serving-cert" not found
Sep 17 18:09:09.880936 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:09.880918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-openshift-service-ca-bundle\") pod \"splitter-graph-9819a-5b6b565754-mn9bp\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") " pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:10.383408 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:10.383379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls\") pod \"splitter-graph-9819a-5b6b565754-mn9bp\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") " pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:10.385954 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:10.385931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls\") pod \"splitter-graph-9819a-5b6b565754-mn9bp\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") " pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:10.630283 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:10.630254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:10.910819 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:10.910791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"]
Sep 17 18:09:10.912652 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:09:10.912622    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ba2c39_06ab_4248_8a2c_9f4e1aea920a.slice/crio-c4656f6a7a2afca1035b397c06d26fc568eb8c827a7b0de7ed0ce9ff20dad915 WatchSource:0}: Error finding container c4656f6a7a2afca1035b397c06d26fc568eb8c827a7b0de7ed0ce9ff20dad915: Status 404 returned error can't find the container with id c4656f6a7a2afca1035b397c06d26fc568eb8c827a7b0de7ed0ce9ff20dad915
Sep 17 18:09:11.480451 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:11.480411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" event={"ID":"91ba2c39-06ab-4248-8a2c-9f4e1aea920a","Type":"ContainerStarted","Data":"9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7"}
Sep 17 18:09:11.480757 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:11.480473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" event={"ID":"91ba2c39-06ab-4248-8a2c-9f4e1aea920a","Type":"ContainerStarted","Data":"c4656f6a7a2afca1035b397c06d26fc568eb8c827a7b0de7ed0ce9ff20dad915"}
Sep 17 18:09:11.480757 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:11.480529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:11.504637 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:11.504588    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" podStartSLOduration=2.504571967 podStartE2EDuration="2.504571967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:11.503024428 +0000 UTC m=+2066.735055248" watchObservedRunningTime="2026-09-17 18:09:11.504571967 +0000 UTC m=+2066.736602786"
Sep 17 18:09:13.094739 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:13.094680    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:13.416978 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:13.416896    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:09:17.491083 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:17.491053    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:18.095295 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:18.095250    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:19.792673 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:19.792635    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"]
Sep 17 18:09:19.793466 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:19.793430    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a" containerID="cri-o://9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7" gracePeriod=30
Sep 17 18:09:19.990033 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:19.989997    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"]
Sep 17 18:09:19.990287 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:19.990224    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container" containerID="cri-o://41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a" gracePeriod=30
Sep 17 18:09:19.990287 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:19.990258    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kube-rbac-proxy" containerID="cri-o://6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822" gracePeriod=30
Sep 17 18:09:20.304989 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.304953    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"]
Sep 17 18:09:20.308529 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.308513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.316217 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.316197    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-892d5-kube-rbac-proxy-sar-config\""
Sep 17 18:09:20.316217 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.316211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-892d5-predictor-serving-cert\""
Sep 17 18:09:20.325413 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.325390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"]
Sep 17 18:09:20.362150 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.362088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.362251 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.362187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9926232-a078-446e-9461-906e8c3ae850-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.362251 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.362243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxg2v\" (UniqueName: \"kubernetes.io/projected/a9926232-a078-446e-9461-906e8c3ae850-kube-api-access-sxg2v\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.462986 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.462957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9926232-a078-446e-9461-906e8c3ae850-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.463094 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.463003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxg2v\" (UniqueName: \"kubernetes.io/projected/a9926232-a078-446e-9461-906e8c3ae850-kube-api-access-sxg2v\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.463094 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.463026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.463172 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:20.463107    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-892d5-predictor-serving-cert: secret "error-404-isvc-892d5-predictor-serving-cert" not found
Sep 17 18:09:20.463172 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:20.463160    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls podName:a9926232-a078-446e-9461-906e8c3ae850 nodeName:}" failed. No retries permitted until 2026-09-17 18:09:20.963146247 +0000 UTC m=+2076.195177044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls") pod "error-404-isvc-892d5-predictor-6659f5749d-6qmvb" (UID: "a9926232-a078-446e-9461-906e8c3ae850") : secret "error-404-isvc-892d5-predictor-serving-cert" not found
Sep 17 18:09:20.463636 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.463614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9926232-a078-446e-9461-906e8c3ae850-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.473578 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.473557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxg2v\" (UniqueName: \"kubernetes.io/projected/a9926232-a078-446e-9461-906e8c3ae850-kube-api-access-sxg2v\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.527741 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.527695    2581 generic.go:356] "Generic (PLEG): container finished" podID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerID="6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822" exitCode=2
Sep 17 18:09:20.527835 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.527768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" event={"ID":"07a60588-4a99-49d3-9ac1-adae9b1f8c69","Type":"ContainerDied","Data":"6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822"}
Sep 17 18:09:20.967307 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.967269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:20.969577 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:20.969559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls\") pod \"error-404-isvc-892d5-predictor-6659f5749d-6qmvb\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") " pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:21.218207 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:21.218117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:21.493480 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:21.493443    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"]
Sep 17 18:09:21.494841 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:09:21.494810    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9926232_a078_446e_9461_906e8c3ae850.slice/crio-8c8190dd0f105ed1fee14fdc9db66108098d7fe833b1dd4e101e41b6139e0ac3 WatchSource:0}: Error finding container 8c8190dd0f105ed1fee14fdc9db66108098d7fe833b1dd4e101e41b6139e0ac3: Status 404 returned error can't find the container with id 8c8190dd0f105ed1fee14fdc9db66108098d7fe833b1dd4e101e41b6139e0ac3
Sep 17 18:09:21.534412 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:21.534380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" event={"ID":"a9926232-a078-446e-9461-906e8c3ae850","Type":"ContainerStarted","Data":"8c8190dd0f105ed1fee14fdc9db66108098d7fe833b1dd4e101e41b6139e0ac3"}
Sep 17 18:09:22.489616 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.489574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:22.543488 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.543450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" event={"ID":"a9926232-a078-446e-9461-906e8c3ae850","Type":"ContainerStarted","Data":"3fa85396e89b121ae5ae81a02927dcfb29a30460e32091d4606d30075b20ff33"}
Sep 17 18:09:22.543636 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.543537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" event={"ID":"a9926232-a078-446e-9461-906e8c3ae850","Type":"ContainerStarted","Data":"2b9bced220f72a5d988d72b4879bcf30d1919be80ce96d58c346da3a4f0ecc0f"}
Sep 17 18:09:22.543636 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.543575    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:22.543803 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.543680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:22.545289 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.545263    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:09:22.565524 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.565473    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podStartSLOduration=2.5654580769999997 podStartE2EDuration="2.565458077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:22.563762782 +0000 UTC m=+2077.795793602" watchObservedRunningTime="2026-09-17 18:09:22.565458077 +0000 UTC m=+2077.797488895"
Sep 17 18:09:22.745802 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.745777    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:09:22.781484 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.781463    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxvb6\" (UniqueName: \"kubernetes.io/projected/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-api-access-fxvb6\") pod \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") "
Sep 17 18:09:22.781607 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.781513    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls\") pod \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") "
Sep 17 18:09:22.781607 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.781539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-rbac-proxy-sar-config\") pod \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\" (UID: \"07a60588-4a99-49d3-9ac1-adae9b1f8c69\") "
Sep 17 18:09:22.781943 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.781915    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07a60588-4a99-49d3-9ac1-adae9b1f8c69" (UID: "07a60588-4a99-49d3-9ac1-adae9b1f8c69"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:22.783560 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.783535    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07a60588-4a99-49d3-9ac1-adae9b1f8c69" (UID: "07a60588-4a99-49d3-9ac1-adae9b1f8c69"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:22.783656 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.783550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-api-access-fxvb6" (OuterVolumeSpecName: "kube-api-access-fxvb6") pod "07a60588-4a99-49d3-9ac1-adae9b1f8c69" (UID: "07a60588-4a99-49d3-9ac1-adae9b1f8c69"). InnerVolumeSpecName "kube-api-access-fxvb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:22.882918 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.882885    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxvb6\" (UniqueName: \"kubernetes.io/projected/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-api-access-fxvb6\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:09:22.882918 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.882916    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07a60588-4a99-49d3-9ac1-adae9b1f8c69-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:09:22.882918 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:22.882926    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07a60588-4a99-49d3-9ac1-adae9b1f8c69-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:09:23.094644 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.094574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:23.416916 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.416839    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:09:23.550066 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.550031    2581 generic.go:356] "Generic (PLEG): container finished" podID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerID="41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a" exitCode=0
Sep 17 18:09:23.550410 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.550091    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"
Sep 17 18:09:23.550410 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.550130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" event={"ID":"07a60588-4a99-49d3-9ac1-adae9b1f8c69","Type":"ContainerDied","Data":"41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a"}
Sep 17 18:09:23.550410 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.550214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf" event={"ID":"07a60588-4a99-49d3-9ac1-adae9b1f8c69","Type":"ContainerDied","Data":"8592dc69c2de48f3388ffc077cc45e8d9860643452e50926cac105ed8f9fbf8d"}
Sep 17 18:09:23.550410 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.550260    2581 scope.go:117] "RemoveContainer" containerID="6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822"
Sep 17 18:09:23.550644 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.550610    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:09:23.559333 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.559309    2581 scope.go:117] "RemoveContainer" containerID="41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a"
Sep 17 18:09:23.567671 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.567652    2581 scope.go:117] "RemoveContainer" containerID="6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822"
Sep 17 18:09:23.567973 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:23.567954    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822\": container with ID starting with 6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822 not found: ID does not exist" containerID="6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822"
Sep 17 18:09:23.568033 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.567982    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822"} err="failed to get container status \"6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822\": rpc error: code = NotFound desc = could not find container \"6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822\": container with ID starting with 6217931f021e22dd5454a0abaa431580833d907012f128040c337b45ca643822 not found: ID does not exist"
Sep 17 18:09:23.568033 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.568001    2581 scope.go:117] "RemoveContainer" containerID="41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a"
Sep 17 18:09:23.568273 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:23.568252    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a\": container with ID starting with 41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a not found: ID does not exist" containerID="41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a"
Sep 17 18:09:23.568326 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.568278    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a"} err="failed to get container status \"41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a\": rpc error: code = NotFound desc = could not find container \"41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a\": container with ID starting with 41162227cf9c69fe04f4f7dcbff2556b81714191d0ab56858e0879011cb74c5a not found: ID does not exist"
Sep 17 18:09:23.573658 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.573634    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"]
Sep 17 18:09:23.578739 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:23.578718    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9819a-predictor-75b9f568db-gpdmf"]
Sep 17 18:09:24.897938 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:24.897910    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" path="/var/lib/kubelet/pods/07a60588-4a99-49d3-9ac1-adae9b1f8c69/volumes"
Sep 17 18:09:25.948203 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:25.948179    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:09:26.009424 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.009394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ce48c7-65f2-4590-917b-423b51760da6-proxy-tls\") pod \"71ce48c7-65f2-4590-917b-423b51760da6\" (UID: \"71ce48c7-65f2-4590-917b-423b51760da6\") "
Sep 17 18:09:26.009558 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.009481    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ce48c7-65f2-4590-917b-423b51760da6-openshift-service-ca-bundle\") pod \"71ce48c7-65f2-4590-917b-423b51760da6\" (UID: \"71ce48c7-65f2-4590-917b-423b51760da6\") "
Sep 17 18:09:26.009765 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.009741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ce48c7-65f2-4590-917b-423b51760da6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "71ce48c7-65f2-4590-917b-423b51760da6" (UID: "71ce48c7-65f2-4590-917b-423b51760da6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:26.011407 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.011387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ce48c7-65f2-4590-917b-423b51760da6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71ce48c7-65f2-4590-917b-423b51760da6" (UID: "71ce48c7-65f2-4590-917b-423b51760da6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:26.110258 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.110197    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ce48c7-65f2-4590-917b-423b51760da6-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:09:26.110258 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.110220    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71ce48c7-65f2-4590-917b-423b51760da6-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:09:26.568207 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.568174    2581 generic.go:356] "Generic (PLEG): container finished" podID="71ce48c7-65f2-4590-917b-423b51760da6" containerID="55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d" exitCode=0
Sep 17 18:09:26.568359 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.568214    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"
Sep 17 18:09:26.568359 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.568252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" event={"ID":"71ce48c7-65f2-4590-917b-423b51760da6","Type":"ContainerDied","Data":"55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d"}
Sep 17 18:09:26.568359 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.568312    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm" event={"ID":"71ce48c7-65f2-4590-917b-423b51760da6","Type":"ContainerDied","Data":"16367af1ca4202414653cbda636fe51113d9d6a91bb94ce968cec1299e259cae"}
Sep 17 18:09:26.568359 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.568332    2581 scope.go:117] "RemoveContainer" containerID="55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d"
Sep 17 18:09:26.577469 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.577453    2581 scope.go:117] "RemoveContainer" containerID="55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d"
Sep 17 18:09:26.577758 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:26.577738    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d\": container with ID starting with 55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d not found: ID does not exist" containerID="55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d"
Sep 17 18:09:26.577835 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.577767    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d"} err="failed to get container status \"55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d\": rpc error: code = NotFound desc = could not find container \"55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d\": container with ID starting with 55d65dca565b4dd0dee5d649556821a7a83413c6e83d70c9d1435e94652b109d not found: ID does not exist"
Sep 17 18:09:26.593370 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.593341    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"]
Sep 17 18:09:26.598294 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.598268    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1e4e7-6f5ffcc466-vpjqm"]
Sep 17 18:09:26.898236 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:26.898172    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ce48c7-65f2-4590-917b-423b51760da6" path="/var/lib/kubelet/pods/71ce48c7-65f2-4590-917b-423b51760da6/volumes"
Sep 17 18:09:27.489831 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:27.489797    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:28.554924 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:28.554895    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:09:28.555425 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:28.555395    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:09:32.489556 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:32.489515    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:32.489948 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:32.489620    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:33.417264 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:33.417225    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:09:37.490249 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:37.490199    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:38.555727 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:38.555663    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:09:42.490142 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:42.490100    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:43.417442 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:43.417409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:09:46.937796 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:46.937765    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:09:46.939947 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:46.939919    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:09:47.489446 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:47.489411    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:09:48.555615 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:48.555579    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:09:49.934914 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:49.934886    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:50.112492 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.112413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-openshift-service-ca-bundle\") pod \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") "
Sep 17 18:09:50.112492 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.112466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls\") pod \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\" (UID: \"91ba2c39-06ab-4248-8a2c-9f4e1aea920a\") "
Sep 17 18:09:50.112817 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.112794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "91ba2c39-06ab-4248-8a2c-9f4e1aea920a" (UID: "91ba2c39-06ab-4248-8a2c-9f4e1aea920a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:50.114557 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.114532    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91ba2c39-06ab-4248-8a2c-9f4e1aea920a" (UID: "91ba2c39-06ab-4248-8a2c-9f4e1aea920a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:50.213510 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.213486    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:09:50.213510 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.213507    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ba2c39-06ab-4248-8a2c-9f4e1aea920a-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:09:50.683679 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.683647    2581 generic.go:356] "Generic (PLEG): container finished" podID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerID="9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7" exitCode=0
Sep 17 18:09:50.683856 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.683692    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"
Sep 17 18:09:50.683856 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.683733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" event={"ID":"91ba2c39-06ab-4248-8a2c-9f4e1aea920a","Type":"ContainerDied","Data":"9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7"}
Sep 17 18:09:50.683856 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.683786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp" event={"ID":"91ba2c39-06ab-4248-8a2c-9f4e1aea920a","Type":"ContainerDied","Data":"c4656f6a7a2afca1035b397c06d26fc568eb8c827a7b0de7ed0ce9ff20dad915"}
Sep 17 18:09:50.683856 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.683806    2581 scope.go:117] "RemoveContainer" containerID="9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7"
Sep 17 18:09:50.692273 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.692254    2581 scope.go:117] "RemoveContainer" containerID="9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7"
Sep 17 18:09:50.692527 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:50.692510    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7\": container with ID starting with 9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7 not found: ID does not exist" containerID="9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7"
Sep 17 18:09:50.692572 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.692535    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7"} err="failed to get container status \"9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7\": rpc error: code = NotFound desc = could not find container \"9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7\": container with ID starting with 9f81f265f953f0b2b010844e41e94fc4c9cbbfe2b7e69d5bb5eae338cc8bdba7 not found: ID does not exist"
Sep 17 18:09:50.707346 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.707321    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"]
Sep 17 18:09:50.709586 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.709567    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9819a-5b6b565754-mn9bp"]
Sep 17 18:09:50.898863 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:50.898833    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" path="/var/lib/kubelet/pods/91ba2c39-06ab-4248-8a2c-9f4e1aea920a/volumes"
Sep 17 18:09:56.175435 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175399    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"]
Sep 17 18:09:56.175818 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175791    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kube-rbac-proxy"
Sep 17 18:09:56.175818 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175804    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kube-rbac-proxy"
Sep 17 18:09:56.175818 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175818    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7"
Sep 17 18:09:56.175928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175824    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7"
Sep 17 18:09:56.175928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175832    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container"
Sep 17 18:09:56.175928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175839    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container"
Sep 17 18:09:56.175928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175850    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a"
Sep 17 18:09:56.175928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175855    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a"
Sep 17 18:09:56.175928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175910    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kserve-container"
Sep 17 18:09:56.175928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175918    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a60588-4a99-49d3-9ac1-adae9b1f8c69" containerName="kube-rbac-proxy"
Sep 17 18:09:56.175928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175924    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ba2c39-06ab-4248-8a2c-9f4e1aea920a" containerName="splitter-graph-9819a"
Sep 17 18:09:56.176174 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.175931    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ce48c7-65f2-4590-917b-423b51760da6" containerName="sequence-graph-1e4e7"
Sep 17 18:09:56.180444 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.180425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:56.185970 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.185950    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-5ec4c-serving-cert\""
Sep 17 18:09:56.185970 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.185960    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-5ec4c-kube-rbac-proxy-sar-config\""
Sep 17 18:09:56.205233 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.205211    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"]
Sep 17 18:09:56.258460 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.258433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls\") pod \"switch-graph-5ec4c-58bffd5c64-54zd4\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") " pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:56.258585 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.258475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0197db6c-fe34-4531-bc88-2854418cc58b-openshift-service-ca-bundle\") pod \"switch-graph-5ec4c-58bffd5c64-54zd4\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") " pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:56.359085 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.359046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls\") pod \"switch-graph-5ec4c-58bffd5c64-54zd4\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") " pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:56.359085 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.359096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0197db6c-fe34-4531-bc88-2854418cc58b-openshift-service-ca-bundle\") pod \"switch-graph-5ec4c-58bffd5c64-54zd4\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") " pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:56.359281 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:56.359177    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-5ec4c-serving-cert: secret "switch-graph-5ec4c-serving-cert" not found
Sep 17 18:09:56.359281 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:09:56.359264    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls podName:0197db6c-fe34-4531-bc88-2854418cc58b nodeName:}" failed. No retries permitted until 2026-09-17 18:09:56.859247735 +0000 UTC m=+2112.091278532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls") pod "switch-graph-5ec4c-58bffd5c64-54zd4" (UID: "0197db6c-fe34-4531-bc88-2854418cc58b") : secret "switch-graph-5ec4c-serving-cert" not found
Sep 17 18:09:56.359723 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.359692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0197db6c-fe34-4531-bc88-2854418cc58b-openshift-service-ca-bundle\") pod \"switch-graph-5ec4c-58bffd5c64-54zd4\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") " pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:56.863777 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.863750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls\") pod \"switch-graph-5ec4c-58bffd5c64-54zd4\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") " pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:56.866096 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:56.866069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls\") pod \"switch-graph-5ec4c-58bffd5c64-54zd4\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") " pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:57.089851 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:57.089811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:57.368306 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:57.368282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"]
Sep 17 18:09:57.370371 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:09:57.370346    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0197db6c_fe34_4531_bc88_2854418cc58b.slice/crio-df47158786183b945565bd9dbae24376bf4cfe9d5bc38bedf6f854b3182f8db2 WatchSource:0}: Error finding container df47158786183b945565bd9dbae24376bf4cfe9d5bc38bedf6f854b3182f8db2: Status 404 returned error can't find the container with id df47158786183b945565bd9dbae24376bf4cfe9d5bc38bedf6f854b3182f8db2
Sep 17 18:09:57.718921 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:57.718833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" event={"ID":"0197db6c-fe34-4531-bc88-2854418cc58b","Type":"ContainerStarted","Data":"87b397fbc4fc3afd2ae676d1d7237b19e6bdc7e1c5b4c1dce6ce918f4bdbb10e"}
Sep 17 18:09:57.718921 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:57.718908    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:09:57.719103 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:57.718936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" event={"ID":"0197db6c-fe34-4531-bc88-2854418cc58b","Type":"ContainerStarted","Data":"df47158786183b945565bd9dbae24376bf4cfe9d5bc38bedf6f854b3182f8db2"}
Sep 17 18:09:57.744618 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:57.744566    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" podStartSLOduration=1.7445482559999999 podStartE2EDuration="1.744548256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:57.742151007 +0000 UTC m=+2112.974181826" watchObservedRunningTime="2026-09-17 18:09:57.744548256 +0000 UTC m=+2112.976579111"
Sep 17 18:09:58.555968 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:09:58.555934    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:10:03.729395 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:03.729360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:10:08.556554 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:08.556524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:10:20.032045 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.032014    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"]
Sep 17 18:10:20.035668 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.035647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.038354 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.038333    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-892d5-serving-cert\""
Sep 17 18:10:20.038454 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.038334    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-892d5-kube-rbac-proxy-sar-config\""
Sep 17 18:10:20.045591 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.045568    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"]
Sep 17 18:10:20.138246 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.138212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-proxy-tls\") pod \"splitter-graph-892d5-75f7f9d99c-4gtmq\" (UID: \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\") " pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.138420 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.138365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-openshift-service-ca-bundle\") pod \"splitter-graph-892d5-75f7f9d99c-4gtmq\" (UID: \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\") " pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.239576 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.239533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-openshift-service-ca-bundle\") pod \"splitter-graph-892d5-75f7f9d99c-4gtmq\" (UID: \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\") " pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.239802 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.239595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-proxy-tls\") pod \"splitter-graph-892d5-75f7f9d99c-4gtmq\" (UID: \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\") " pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.240195 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.240171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-openshift-service-ca-bundle\") pod \"splitter-graph-892d5-75f7f9d99c-4gtmq\" (UID: \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\") " pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.242012 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.241988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-proxy-tls\") pod \"splitter-graph-892d5-75f7f9d99c-4gtmq\" (UID: \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\") " pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.347282 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.347195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.620940 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.620909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"]
Sep 17 18:10:20.622620 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:10:20.622594    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d74c01a_17ec_4e13_b0dd_c13940b68d2a.slice/crio-28fe2eb83ba98132d3c926b8f3dd66eddad38b959f8780656a3058ea035e6e6e WatchSource:0}: Error finding container 28fe2eb83ba98132d3c926b8f3dd66eddad38b959f8780656a3058ea035e6e6e: Status 404 returned error can't find the container with id 28fe2eb83ba98132d3c926b8f3dd66eddad38b959f8780656a3058ea035e6e6e
Sep 17 18:10:20.830420 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.830381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" event={"ID":"7d74c01a-17ec-4e13-b0dd-c13940b68d2a","Type":"ContainerStarted","Data":"6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42"}
Sep 17 18:10:20.830608 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.830442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" event={"ID":"7d74c01a-17ec-4e13-b0dd-c13940b68d2a","Type":"ContainerStarted","Data":"28fe2eb83ba98132d3c926b8f3dd66eddad38b959f8780656a3058ea035e6e6e"}
Sep 17 18:10:20.830608 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.830466    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:10:20.854618 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:20.854561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" podStartSLOduration=0.854545496 podStartE2EDuration="854.545496ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:20.853256408 +0000 UTC m=+2136.085287228" watchObservedRunningTime="2026-09-17 18:10:20.854545496 +0000 UTC m=+2136.086576344"
Sep 17 18:10:26.841464 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:10:26.841432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:14:46.967942 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:14:46.967910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:14:46.970508 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:14:46.970488    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:18:34.518843 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:34.518811    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"]
Sep 17 18:18:34.519521 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:34.519007    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5" containerID="cri-o://6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42" gracePeriod=30
Sep 17 18:18:34.683379 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:34.683339    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"]
Sep 17 18:18:34.683629 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:34.683596    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container" containerID="cri-o://2b9bced220f72a5d988d72b4879bcf30d1919be80ce96d58c346da3a4f0ecc0f" gracePeriod=30
Sep 17 18:18:34.683739 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:34.683615    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kube-rbac-proxy" containerID="cri-o://3fa85396e89b121ae5ae81a02927dcfb29a30460e32091d4606d30075b20ff33" gracePeriod=30
Sep 17 18:18:35.148678 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:35.148590    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9926232-a078-446e-9461-906e8c3ae850" containerID="3fa85396e89b121ae5ae81a02927dcfb29a30460e32091d4606d30075b20ff33" exitCode=2
Sep 17 18:18:35.148678 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:35.148651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" event={"ID":"a9926232-a078-446e-9461-906e8c3ae850","Type":"ContainerDied","Data":"3fa85396e89b121ae5ae81a02927dcfb29a30460e32091d4606d30075b20ff33"}
Sep 17 18:18:36.839548 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:36.839495    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:18:37.159936 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.159906    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9926232-a078-446e-9461-906e8c3ae850" containerID="2b9bced220f72a5d988d72b4879bcf30d1919be80ce96d58c346da3a4f0ecc0f" exitCode=0
Sep 17 18:18:37.160064 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.159983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" event={"ID":"a9926232-a078-446e-9461-906e8c3ae850","Type":"ContainerDied","Data":"2b9bced220f72a5d988d72b4879bcf30d1919be80ce96d58c346da3a4f0ecc0f"}
Sep 17 18:18:37.242426 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.242403    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:18:37.425310 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.425240    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9926232-a078-446e-9461-906e8c3ae850-kube-rbac-proxy-sar-config\") pod \"a9926232-a078-446e-9461-906e8c3ae850\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") "
Sep 17 18:18:37.425310 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.425278    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxg2v\" (UniqueName: \"kubernetes.io/projected/a9926232-a078-446e-9461-906e8c3ae850-kube-api-access-sxg2v\") pod \"a9926232-a078-446e-9461-906e8c3ae850\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") "
Sep 17 18:18:37.425473 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.425330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls\") pod \"a9926232-a078-446e-9461-906e8c3ae850\" (UID: \"a9926232-a078-446e-9461-906e8c3ae850\") "
Sep 17 18:18:37.425652 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.425620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9926232-a078-446e-9461-906e8c3ae850-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9926232-a078-446e-9461-906e8c3ae850" (UID: "a9926232-a078-446e-9461-906e8c3ae850"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:37.427380 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.427333    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9926232-a078-446e-9461-906e8c3ae850" (UID: "a9926232-a078-446e-9461-906e8c3ae850"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:37.427483 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.427422    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9926232-a078-446e-9461-906e8c3ae850-kube-api-access-sxg2v" (OuterVolumeSpecName: "kube-api-access-sxg2v") pod "a9926232-a078-446e-9461-906e8c3ae850" (UID: "a9926232-a078-446e-9461-906e8c3ae850"). InnerVolumeSpecName "kube-api-access-sxg2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:37.525928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.525902    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9926232-a078-446e-9461-906e8c3ae850-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.525928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.525928    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9926232-a078-446e-9461-906e8c3ae850-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:18:37.526094 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:37.525943    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxg2v\" (UniqueName: \"kubernetes.io/projected/a9926232-a078-446e-9461-906e8c3ae850-kube-api-access-sxg2v\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:18:38.166889 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:38.166860    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"
Sep 17 18:18:38.167293 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:38.166893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb" event={"ID":"a9926232-a078-446e-9461-906e8c3ae850","Type":"ContainerDied","Data":"8c8190dd0f105ed1fee14fdc9db66108098d7fe833b1dd4e101e41b6139e0ac3"}
Sep 17 18:18:38.167293 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:38.166988    2581 scope.go:117] "RemoveContainer" containerID="3fa85396e89b121ae5ae81a02927dcfb29a30460e32091d4606d30075b20ff33"
Sep 17 18:18:38.179632 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:38.179603    2581 scope.go:117] "RemoveContainer" containerID="2b9bced220f72a5d988d72b4879bcf30d1919be80ce96d58c346da3a4f0ecc0f"
Sep 17 18:18:38.190922 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:38.190894    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"]
Sep 17 18:18:38.195848 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:38.195822    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-892d5-predictor-6659f5749d-6qmvb"]
Sep 17 18:18:38.898990 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:38.898949    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9926232-a078-446e-9461-906e8c3ae850" path="/var/lib/kubelet/pods/a9926232-a078-446e-9461-906e8c3ae850/volumes"
Sep 17 18:18:41.839875 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:41.839828    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:18:46.839105 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:46.839068    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:18:46.839526 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:46.839175    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:18:51.839069 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:51.839031    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:18:56.839116 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:18:56.839075    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:19:01.839093 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:01.839051    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:19:04.666618 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:04.666591    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:19:04.726927 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:04.726898    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-openshift-service-ca-bundle\") pod \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\" (UID: \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\") "
Sep 17 18:19:04.727125 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:04.726968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-proxy-tls\") pod \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\" (UID: \"7d74c01a-17ec-4e13-b0dd-c13940b68d2a\") "
Sep 17 18:19:04.727274 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:04.727249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7d74c01a-17ec-4e13-b0dd-c13940b68d2a" (UID: "7d74c01a-17ec-4e13-b0dd-c13940b68d2a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:04.728921 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:04.728897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d74c01a-17ec-4e13-b0dd-c13940b68d2a" (UID: "7d74c01a-17ec-4e13-b0dd-c13940b68d2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:04.827755 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:04.827662    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:19:04.827755 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:04.827691    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d74c01a-17ec-4e13-b0dd-c13940b68d2a-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:19:05.290344 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.290305    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerID="6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42" exitCode=0
Sep 17 18:19:05.290550 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.290370    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"
Sep 17 18:19:05.290550 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.290391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" event={"ID":"7d74c01a-17ec-4e13-b0dd-c13940b68d2a","Type":"ContainerDied","Data":"6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42"}
Sep 17 18:19:05.290550 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.290450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq" event={"ID":"7d74c01a-17ec-4e13-b0dd-c13940b68d2a","Type":"ContainerDied","Data":"28fe2eb83ba98132d3c926b8f3dd66eddad38b959f8780656a3058ea035e6e6e"}
Sep 17 18:19:05.290550 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.290471    2581 scope.go:117] "RemoveContainer" containerID="6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42"
Sep 17 18:19:05.299064 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.299043    2581 scope.go:117] "RemoveContainer" containerID="6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42"
Sep 17 18:19:05.299384 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:19:05.299348    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42\": container with ID starting with 6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42 not found: ID does not exist" containerID="6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42"
Sep 17 18:19:05.299454 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.299391    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42"} err="failed to get container status \"6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42\": rpc error: code = NotFound desc = could not find container \"6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42\": container with ID starting with 6bab17b006ab2760e48081bf8165617ccbae30379cb4fc4e46267ac02dbddb42 not found: ID does not exist"
Sep 17 18:19:05.310050 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.310025    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"]
Sep 17 18:19:05.314861 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:05.314838    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-892d5-75f7f9d99c-4gtmq"]
Sep 17 18:19:06.899257 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:06.899221    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" path="/var/lib/kubelet/pods/7d74c01a-17ec-4e13-b0dd-c13940b68d2a/volumes"
Sep 17 18:19:46.996846 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:46.996815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:19:47.000095 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:19:47.000076    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:24:47.032982 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:24:47.032945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:24:47.040132 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:24:47.040107    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:26:15.333219 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:15.333188    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"]
Sep 17 18:26:15.333754 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:15.333410    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c" containerID="cri-o://87b397fbc4fc3afd2ae676d1d7237b19e6bdc7e1c5b4c1dce6ce918f4bdbb10e" gracePeriod=30
Sep 17 18:26:15.496285 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:15.496253    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"]
Sep 17 18:26:15.496524 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:15.496479    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container" containerID="cri-o://93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95" gracePeriod=30
Sep 17 18:26:15.496605 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:15.496539    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kube-rbac-proxy" containerID="cri-o://3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859" gracePeriod=30
Sep 17 18:26:16.306787 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:16.306750    2581 generic.go:356] "Generic (PLEG): container finished" podID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerID="3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859" exitCode=2
Sep 17 18:26:16.306962 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:16.306818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" event={"ID":"85aed06b-af8a-410d-9c0d-a76d1722e3fd","Type":"ContainerDied","Data":"3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859"}
Sep 17 18:26:18.133493 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.133466    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:26:18.280352 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.280328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fznh\" (UniqueName: \"kubernetes.io/projected/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-api-access-4fznh\") pod \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") "
Sep 17 18:26:18.280492 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.280360    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-rbac-proxy-sar-config\") pod \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") "
Sep 17 18:26:18.280492 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.280402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85aed06b-af8a-410d-9c0d-a76d1722e3fd-proxy-tls\") pod \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\" (UID: \"85aed06b-af8a-410d-9c0d-a76d1722e3fd\") "
Sep 17 18:26:18.280734 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.280689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85aed06b-af8a-410d-9c0d-a76d1722e3fd" (UID: "85aed06b-af8a-410d-9c0d-a76d1722e3fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:18.282405 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.282379    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-api-access-4fznh" (OuterVolumeSpecName: "kube-api-access-4fznh") pod "85aed06b-af8a-410d-9c0d-a76d1722e3fd" (UID: "85aed06b-af8a-410d-9c0d-a76d1722e3fd"). InnerVolumeSpecName "kube-api-access-4fznh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:18.282494 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.282469    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85aed06b-af8a-410d-9c0d-a76d1722e3fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85aed06b-af8a-410d-9c0d-a76d1722e3fd" (UID: "85aed06b-af8a-410d-9c0d-a76d1722e3fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:18.317291 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.317262    2581 generic.go:356] "Generic (PLEG): container finished" podID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerID="93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95" exitCode=0
Sep 17 18:26:18.317377 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.317296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" event={"ID":"85aed06b-af8a-410d-9c0d-a76d1722e3fd","Type":"ContainerDied","Data":"93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95"}
Sep 17 18:26:18.317377 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.317305    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"
Sep 17 18:26:18.317377 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.317342    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv" event={"ID":"85aed06b-af8a-410d-9c0d-a76d1722e3fd","Type":"ContainerDied","Data":"c7798913132220b29cf6546e9c2b52b7aba740b9d64f90f515b0b27dbea98f88"}
Sep 17 18:26:18.317377 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.317369    2581 scope.go:117] "RemoveContainer" containerID="3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859"
Sep 17 18:26:18.326075 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.326060    2581 scope.go:117] "RemoveContainer" containerID="93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95"
Sep 17 18:26:18.333867 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.333844    2581 scope.go:117] "RemoveContainer" containerID="3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859"
Sep 17 18:26:18.334119 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:26:18.334098    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859\": container with ID starting with 3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859 not found: ID does not exist" containerID="3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859"
Sep 17 18:26:18.334184 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.334126    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859"} err="failed to get container status \"3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859\": rpc error: code = NotFound desc = could not find container \"3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859\": container with ID starting with 3985d17701ec6da33c1e703fb02951da7fdb4ca3ae33967273d4fb2772983859 not found: ID does not exist"
Sep 17 18:26:18.334184 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.334142    2581 scope.go:117] "RemoveContainer" containerID="93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95"
Sep 17 18:26:18.334358 ip-10-0-133-107 kubenswrapper[2581]: E0917 18:26:18.334342    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95\": container with ID starting with 93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95 not found: ID does not exist" containerID="93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95"
Sep 17 18:26:18.334400 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.334362    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95"} err="failed to get container status \"93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95\": rpc error: code = NotFound desc = could not find container \"93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95\": container with ID starting with 93403566cf97e148887d0b48b701c5979ac6c3e77c6696e41687ad5eb74f8a95 not found: ID does not exist"
Sep 17 18:26:18.339868 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.339846    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"]
Sep 17 18:26:18.344370 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.344349    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5ec4c-predictor-68b6c44d66-hmwnv"]
Sep 17 18:26:18.381109 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.381086    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fznh\" (UniqueName: \"kubernetes.io/projected/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-api-access-4fznh\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:26:18.381109 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.381105    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85aed06b-af8a-410d-9c0d-a76d1722e3fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:26:18.381210 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.381115    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85aed06b-af8a-410d-9c0d-a76d1722e3fd-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:26:18.728470 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.728390    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:26:18.898677 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:18.898642    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" path="/var/lib/kubelet/pods/85aed06b-af8a-410d-9c0d-a76d1722e3fd/volumes"
Sep 17 18:26:23.728364 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:23.728322    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:26:28.728165 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:28.728130    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:26:28.728522 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:28.728221    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:26:32.080417 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:32.080378    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:32.989169 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:32.989130    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:33.727964 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:33.727927    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:26:33.815873 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:33.815845    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:34.620566 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:34.620538    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:35.426358 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:35.426320    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:36.236153 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:36.236123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:37.048505 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:37.048477    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:37.881005 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:37.880947    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:38.700590 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:38.700563    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:38.727669 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:38.727637    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:26:39.506217 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:39.506180    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:40.352532 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:40.352502    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:41.173913 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:41.173863    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:41.987485 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:41.987452    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:42.799802 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:42.799774    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:43.641392 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:43.641363    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5ec4c-58bffd5c64-54zd4_0197db6c-fe34-4531-bc88-2854418cc58b/switch-graph-5ec4c/0.log"
Sep 17 18:26:43.727870 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:43.727830    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:26:45.464398 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.464368    2581 generic.go:356] "Generic (PLEG): container finished" podID="0197db6c-fe34-4531-bc88-2854418cc58b" containerID="87b397fbc4fc3afd2ae676d1d7237b19e6bdc7e1c5b4c1dce6ce918f4bdbb10e" exitCode=0
Sep 17 18:26:45.464692 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.464428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" event={"ID":"0197db6c-fe34-4531-bc88-2854418cc58b","Type":"ContainerDied","Data":"87b397fbc4fc3afd2ae676d1d7237b19e6bdc7e1c5b4c1dce6ce918f4bdbb10e"}
Sep 17 18:26:45.464692 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.464467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4" event={"ID":"0197db6c-fe34-4531-bc88-2854418cc58b","Type":"ContainerDied","Data":"df47158786183b945565bd9dbae24376bf4cfe9d5bc38bedf6f854b3182f8db2"}
Sep 17 18:26:45.464692 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.464489    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df47158786183b945565bd9dbae24376bf4cfe9d5bc38bedf6f854b3182f8db2"
Sep 17 18:26:45.475858 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.475838    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:26:45.584965 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.584939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls\") pod \"0197db6c-fe34-4531-bc88-2854418cc58b\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") "
Sep 17 18:26:45.585078 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.584971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0197db6c-fe34-4531-bc88-2854418cc58b-openshift-service-ca-bundle\") pod \"0197db6c-fe34-4531-bc88-2854418cc58b\" (UID: \"0197db6c-fe34-4531-bc88-2854418cc58b\") "
Sep 17 18:26:45.585349 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.585316    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0197db6c-fe34-4531-bc88-2854418cc58b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0197db6c-fe34-4531-bc88-2854418cc58b" (UID: "0197db6c-fe34-4531-bc88-2854418cc58b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:45.586914 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.586894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0197db6c-fe34-4531-bc88-2854418cc58b" (UID: "0197db6c-fe34-4531-bc88-2854418cc58b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:45.686233 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.686209    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0197db6c-fe34-4531-bc88-2854418cc58b-proxy-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:26:45.686233 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:45.686231    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0197db6c-fe34-4531-bc88-2854418cc58b-openshift-service-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
Sep 17 18:26:46.469494 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:46.469462    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"
Sep 17 18:26:46.496499 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:46.496467    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"]
Sep 17 18:26:46.504081 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:46.504053    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5ec4c-58bffd5c64-54zd4"]
Sep 17 18:26:46.900020 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:46.899992    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" path="/var/lib/kubelet/pods/0197db6c-fe34-4531-bc88-2854418cc58b/volumes"
Sep 17 18:26:47.187111 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.187044    2581 scope.go:117] "RemoveContainer" containerID="87b397fbc4fc3afd2ae676d1d7237b19e6bdc7e1c5b4c1dce6ce918f4bdbb10e"
Sep 17 18:26:47.803655 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.803625    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ffk4x/must-gather-d4p7b"]
Sep 17 18:26:47.804008 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.803964    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kube-rbac-proxy"
Sep 17 18:26:47.804008 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.803976    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kube-rbac-proxy"
Sep 17 18:26:47.804008 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.803984    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c"
Sep 17 18:26:47.804008 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.803990    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c"
Sep 17 18:26:47.804008 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804004    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container"
Sep 17 18:26:47.804008 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804009    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804016    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804021    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804030    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804035    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804044    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kube-rbac-proxy"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804050    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kube-rbac-proxy"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804100    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kube-rbac-proxy"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804108    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0197db6c-fe34-4531-bc88-2854418cc58b" containerName="switch-graph-5ec4c"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804116    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="85aed06b-af8a-410d-9c0d-a76d1722e3fd" containerName="kserve-container"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804122    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kserve-container"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804127    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9926232-a078-446e-9461-906e8c3ae850" containerName="kube-rbac-proxy"
Sep 17 18:26:47.804201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.804134    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d74c01a-17ec-4e13-b0dd-c13940b68d2a" containerName="splitter-graph-892d5"
Sep 17 18:26:47.808306 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.808288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ffk4x/must-gather-d4p7b"
Sep 17 18:26:47.810868 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.810844    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ffk4x\"/\"default-dockercfg-lw4rk\""
Sep 17 18:26:47.810988 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.810874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ffk4x\"/\"openshift-service-ca.crt\""
Sep 17 18:26:47.810988 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.810934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ffk4x\"/\"kube-root-ca.crt\""
Sep 17 18:26:47.817816 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.817794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ffk4x/must-gather-d4p7b"]
Sep 17 18:26:47.900046 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.900021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b78kp\" (UniqueName: \"kubernetes.io/projected/4253691c-57e3-4410-ba34-282299752eba-kube-api-access-b78kp\") pod \"must-gather-d4p7b\" (UID: \"4253691c-57e3-4410-ba34-282299752eba\") " pod="openshift-must-gather-ffk4x/must-gather-d4p7b"
Sep 17 18:26:47.900144 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:47.900068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4253691c-57e3-4410-ba34-282299752eba-must-gather-output\") pod \"must-gather-d4p7b\" (UID: \"4253691c-57e3-4410-ba34-282299752eba\") " pod="openshift-must-gather-ffk4x/must-gather-d4p7b"
Sep 17 18:26:48.001344 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:48.001321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4253691c-57e3-4410-ba34-282299752eba-must-gather-output\") pod \"must-gather-d4p7b\" (UID: \"4253691c-57e3-4410-ba34-282299752eba\") " pod="openshift-must-gather-ffk4x/must-gather-d4p7b"
Sep 17 18:26:48.001439 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:48.001387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b78kp\" (UniqueName: \"kubernetes.io/projected/4253691c-57e3-4410-ba34-282299752eba-kube-api-access-b78kp\") pod \"must-gather-d4p7b\" (UID: \"4253691c-57e3-4410-ba34-282299752eba\") " pod="openshift-must-gather-ffk4x/must-gather-d4p7b"
Sep 17 18:26:48.001620 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:48.001603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4253691c-57e3-4410-ba34-282299752eba-must-gather-output\") pod \"must-gather-d4p7b\" (UID: \"4253691c-57e3-4410-ba34-282299752eba\") " pod="openshift-must-gather-ffk4x/must-gather-d4p7b"
Sep 17 18:26:48.012293 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:48.012268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b78kp\" (UniqueName: \"kubernetes.io/projected/4253691c-57e3-4410-ba34-282299752eba-kube-api-access-b78kp\") pod \"must-gather-d4p7b\" (UID: \"4253691c-57e3-4410-ba34-282299752eba\") " pod="openshift-must-gather-ffk4x/must-gather-d4p7b"
Sep 17 18:26:48.128026 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:48.127977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ffk4x/must-gather-d4p7b"
Sep 17 18:26:48.398161 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:48.398063    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ffk4x/must-gather-d4p7b"]
Sep 17 18:26:48.400518 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:26:48.400489    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4253691c_57e3_4410_ba34_282299752eba.slice/crio-74aef9f2829d2abf78202b9f72787078652f94bedd9779e57b905561fc9ef332 WatchSource:0}: Error finding container 74aef9f2829d2abf78202b9f72787078652f94bedd9779e57b905561fc9ef332: Status 404 returned error can't find the container with id 74aef9f2829d2abf78202b9f72787078652f94bedd9779e57b905561fc9ef332
Sep 17 18:26:48.402567 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:48.402549    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:26:48.480045 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:48.480017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ffk4x/must-gather-d4p7b" event={"ID":"4253691c-57e3-4410-ba34-282299752eba","Type":"ContainerStarted","Data":"74aef9f2829d2abf78202b9f72787078652f94bedd9779e57b905561fc9ef332"}
Sep 17 18:26:49.486986 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:49.486905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ffk4x/must-gather-d4p7b" event={"ID":"4253691c-57e3-4410-ba34-282299752eba","Type":"ContainerStarted","Data":"e6c7d18804fb45e5042a143e5bdf5e1cc9822385a6bed899f525cddfc7311574"}
Sep 17 18:26:49.486986 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:49.486950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ffk4x/must-gather-d4p7b" event={"ID":"4253691c-57e3-4410-ba34-282299752eba","Type":"ContainerStarted","Data":"0350a37ab5b0cac6ffd98043368812083df0e3559457e3566d5e72499f9c28e1"}
Sep 17 18:26:49.505276 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:49.505217    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ffk4x/must-gather-d4p7b" podStartSLOduration=1.685899212 podStartE2EDuration="2.505195029s" totalImagesPullingTime="819.295817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:49.504348256 +0000 UTC m=+3124.736379075" watchObservedRunningTime="2026-09-17 18:26:49.505195029 +0000 UTC m=+3124.737225850"
Sep 17 18:26:50.773789 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:50.773757    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2bl7r_6d832701-c936-4fba-b54d-df1ed6594c63/global-pull-secret-syncer/0.log"
Sep 17 18:26:50.952864 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:50.952835    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bfjrr_0fc71244-a845-45e1-b684-12cf8dd012f6/konnectivity-agent/0.log"
Sep 17 18:26:51.077176 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:51.077094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-107.ec2.internal_a0a53354102c1933fab32ef2c66f3050/haproxy/0.log"
Sep 17 18:26:55.212612 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.212486    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bc3705c-084c-4c36-a694-382e2ae9eeb3/alertmanager/0.log"
Sep 17 18:26:55.256782 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.256745    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bc3705c-084c-4c36-a694-382e2ae9eeb3/config-reloader/0.log"
Sep 17 18:26:55.292226 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.292191    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bc3705c-084c-4c36-a694-382e2ae9eeb3/kube-rbac-proxy-web/0.log"
Sep 17 18:26:55.324100 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.324064    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bc3705c-084c-4c36-a694-382e2ae9eeb3/kube-rbac-proxy/0.log"
Sep 17 18:26:55.353989 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.353959    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bc3705c-084c-4c36-a694-382e2ae9eeb3/kube-rbac-proxy-metric/0.log"
Sep 17 18:26:55.385591 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.385560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bc3705c-084c-4c36-a694-382e2ae9eeb3/prom-label-proxy/0.log"
Sep 17 18:26:55.415120 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.415091    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bc3705c-084c-4c36-a694-382e2ae9eeb3/init-config-reloader/0.log"
Sep 17 18:26:55.466191 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.466117    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-xlltt_68ec1a62-564e-403e-a5a3-85179a62f64f/cluster-monitoring-operator/0.log"
Sep 17 18:26:55.627559 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.627531    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-vn6st_3e46f46e-b59b-483f-8647-3786c1430d0c/monitoring-plugin/0.log"
Sep 17 18:26:55.845341 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.845311    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b2pr8_90919883-fc9e-41e9-b4b0-14e7e8b961e8/node-exporter/0.log"
Sep 17 18:26:55.875438 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.875408    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b2pr8_90919883-fc9e-41e9-b4b0-14e7e8b961e8/kube-rbac-proxy/0.log"
Sep 17 18:26:55.900207 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:55.900178    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b2pr8_90919883-fc9e-41e9-b4b0-14e7e8b961e8/init-textfile/0.log"
Sep 17 18:26:56.242120 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:56.242043    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-gwsxs_72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf/prometheus-operator/0.log"
Sep 17 18:26:56.281682 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:56.281651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-gwsxs_72bf23c8-91d6-4e7f-bd8e-94ea1acf8cbf/kube-rbac-proxy/0.log"
Sep 17 18:26:56.355547 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:56.355518    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f4dcbbfb9-685zg_722a1816-b9f5-4efe-b5ee-a614e6422524/telemeter-client/0.log"
Sep 17 18:26:56.410412 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:56.410387    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f4dcbbfb9-685zg_722a1816-b9f5-4efe-b5ee-a614e6422524/reload/0.log"
Sep 17 18:26:56.455376 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:56.455349    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f4dcbbfb9-685zg_722a1816-b9f5-4efe-b5ee-a614e6422524/kube-rbac-proxy/0.log"
Sep 17 18:26:57.856330 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:57.856246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-b9f8t_26a0a5e6-ffa0-43c2-be9c-0e1cd510bac3/networking-console-plugin/0.log"
Sep 17 18:26:58.543081 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.543046    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"]
Sep 17 18:26:58.548481 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.548449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.555465 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.555208    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"]
Sep 17 18:26:58.601993 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.601957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-lib-modules\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.602187 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.602003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwkv\" (UniqueName: \"kubernetes.io/projected/a6ea356a-0117-4b52-8815-387a65d4b111-kube-api-access-tdwkv\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.602187 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.602086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-proc\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.602187 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.602107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-sys\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.602363 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.602208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-podres\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703275 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-lib-modules\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703483 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwkv\" (UniqueName: \"kubernetes.io/projected/a6ea356a-0117-4b52-8815-387a65d4b111-kube-api-access-tdwkv\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703483 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-proc\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703483 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-lib-modules\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703483 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-sys\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703483 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-sys\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703483 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-proc\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703775 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-podres\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.703775 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.703601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a6ea356a-0117-4b52-8815-387a65d4b111-podres\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.717201 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.717166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwkv\" (UniqueName: \"kubernetes.io/projected/a6ea356a-0117-4b52-8815-387a65d4b111-kube-api-access-tdwkv\") pod \"perf-node-gather-daemonset-xkc7c\" (UID: \"a6ea356a-0117-4b52-8815-387a65d4b111\") " pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.852469 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.852390    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5dc75565b7-v4kdl_ef1a9e76-99c3-4599-a28c-96f45fd0eac1/console/0.log"
Sep 17 18:26:58.862928 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.862896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:58.903351 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:58.903313    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-cprrn_4dbeed9c-f53c-4e22-afc0-a7549595158b/download-server/0.log"
Sep 17 18:26:59.147719 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:59.147595    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"]
Sep 17 18:26:59.150729 ip-10-0-133-107 kubenswrapper[2581]: W0917 18:26:59.150672    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda6ea356a_0117_4b52_8815_387a65d4b111.slice/crio-b42797da8fd020f7b73b67421ae4d63ca332e5ab31f940ae2d2d6e1a3184fbf7 WatchSource:0}: Error finding container b42797da8fd020f7b73b67421ae4d63ca332e5ab31f940ae2d2d6e1a3184fbf7: Status 404 returned error can't find the container with id b42797da8fd020f7b73b67421ae4d63ca332e5ab31f940ae2d2d6e1a3184fbf7
Sep 17 18:26:59.484435 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:59.484351    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-xzc7q_975b9941-2cf8-4a56-901c-72e4d0a0d7f8/volume-data-source-validator/0.log"
Sep 17 18:26:59.559694 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:59.559647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c" event={"ID":"a6ea356a-0117-4b52-8815-387a65d4b111","Type":"ContainerStarted","Data":"e599b187bf3750b9d37bfb1ccb4ac2ae57562bcb058d26edfcfc6966e7261842"}
Sep 17 18:26:59.559914 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:59.559738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c" event={"ID":"a6ea356a-0117-4b52-8815-387a65d4b111","Type":"ContainerStarted","Data":"b42797da8fd020f7b73b67421ae4d63ca332e5ab31f940ae2d2d6e1a3184fbf7"}
Sep 17 18:26:59.559914 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:59.559877    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:26:59.578727 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:26:59.578653    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c" podStartSLOduration=1.578632926 podStartE2EDuration="1.578632926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:59.577480403 +0000 UTC m=+3134.809511238" watchObservedRunningTime="2026-09-17 18:26:59.578632926 +0000 UTC m=+3134.810663746"
Sep 17 18:27:00.520291 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:00.520260    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmpdc_ecbf5bec-f51d-44ff-ba63-a47812c3b733/dns/0.log"
Sep 17 18:27:00.547258 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:00.547227    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmpdc_ecbf5bec-f51d-44ff-ba63-a47812c3b733/kube-rbac-proxy/0.log"
Sep 17 18:27:00.633356 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:00.633329    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbsv8_753a888b-2139-4ac2-afbf-2085bf6df478/dns-node-resolver/0.log"
Sep 17 18:27:01.079783 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:01.079728    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b6df65cc4-jd6g9_e7307cdc-4814-4e1a-803c-1aad13e3ddb1/registry/0.log"
Sep 17 18:27:01.106229 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:01.106195    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b6df65cc4-jd6g9_e7307cdc-4814-4e1a-803c-1aad13e3ddb1/registry/1.log"
Sep 17 18:27:01.158596 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:01.158563    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jjc62_fa4430ae-bc53-43aa-bc0e-77cb787e0d32/node-ca/0.log"
Sep 17 18:27:01.927788 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:01.927755    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8559889cdb-wd9h9_71a9e678-1ce7-4e6e-b8bc-6140a7984222/router/0.log"
Sep 17 18:27:02.289468 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:02.289432    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9kcf6_27b7bd4d-0c64-42ff-8ad6-a5a9d5a9ee71/serve-healthcheck-canary/0.log"
Sep 17 18:27:02.750577 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:02.750532    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-dfd2h_51a506c7-09c6-4921-91b2-fd8ea4a044c8/insights-operator/0.log"
Sep 17 18:27:02.752006 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:02.751973    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-dfd2h_51a506c7-09c6-4921-91b2-fd8ea4a044c8/insights-operator/1.log"
Sep 17 18:27:02.786215 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:02.786192    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dz2vf_44e82ca8-6bf7-44a5-96aa-3c28ba930259/kube-rbac-proxy/0.log"
Sep 17 18:27:02.811417 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:02.811383    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dz2vf_44e82ca8-6bf7-44a5-96aa-3c28ba930259/exporter/0.log"
Sep 17 18:27:02.839064 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:02.839037    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dz2vf_44e82ca8-6bf7-44a5-96aa-3c28ba930259/extractor/0.log"
Sep 17 18:27:05.575853 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:05.575827    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ffk4x/perf-node-gather-daemonset-xkc7c"
Sep 17 18:27:05.580014 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:05.579979    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7f654675b-cc47q_f33aad09-e3ee-4bfb-b420-9f6765e40139/manager/0.log"
Sep 17 18:27:05.667244 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:05.667215    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-wm4k8_e860015c-8390-41bd-a4a1-e191c1bc3faa/server/0.log"
Sep 17 18:27:05.840965 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:05.840885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-vlvp7_257b18e4-d825-4749-a2d5-229d00cf61c4/manager/0.log"
Sep 17 18:27:05.959687 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:05.959655    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-69zb9_e6210316-d84d-490b-b92b-7efa0c61438c/seaweedfs/0.log"
Sep 17 18:27:12.134265 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:12.134232    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krjlf_91affe17-634c-4ac8-b1a7-b60485e21db4/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:27:12.163559 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:12.163527    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krjlf_91affe17-634c-4ac8-b1a7-b60485e21db4/egress-router-binary-copy/0.log"
Sep 17 18:27:12.191272 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:12.191248    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krjlf_91affe17-634c-4ac8-b1a7-b60485e21db4/cni-plugins/0.log"
Sep 17 18:27:12.215983 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:12.215955    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krjlf_91affe17-634c-4ac8-b1a7-b60485e21db4/bond-cni-plugin/0.log"
Sep 17 18:27:12.252766 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:12.252736    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krjlf_91affe17-634c-4ac8-b1a7-b60485e21db4/routeoverride-cni/0.log"
Sep 17 18:27:12.285498 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:12.285468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krjlf_91affe17-634c-4ac8-b1a7-b60485e21db4/whereabouts-cni-bincopy/0.log"
Sep 17 18:27:12.321018 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:12.320987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krjlf_91affe17-634c-4ac8-b1a7-b60485e21db4/whereabouts-cni/0.log"
Sep 17 18:27:12.964673 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:12.964631    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jbqnx_546c6e44-f2b9-45e6-bdb9-6682f3f061d9/kube-multus/0.log"
Sep 17 18:27:13.092104 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:13.092062    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kc6h8_b665814a-b3dd-4590-9dbf-63559f1f5681/network-metrics-daemon/0.log"
Sep 17 18:27:13.116663 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:13.116632    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kc6h8_b665814a-b3dd-4590-9dbf-63559f1f5681/kube-rbac-proxy/0.log"
Sep 17 18:27:14.235342 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.235293    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-controller/0.log"
Sep 17 18:27:14.255277 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.255231    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/1.log"
Sep 17 18:27:14.255277 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.255256    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovn-acl-logging/0.log"
Sep 17 18:27:14.277923 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.277899    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/kube-rbac-proxy-node/0.log"
Sep 17 18:27:14.306821 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.306770    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:27:14.332755 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.332722    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/northd/0.log"
Sep 17 18:27:14.358652 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.358623    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/nbdb/0.log"
Sep 17 18:27:14.385184 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.385155    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/sbdb/0.log"
Sep 17 18:27:14.451566 ip-10-0-133-107 kubenswrapper[2581]: I0917 18:27:14.451526    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ch8qr_fb65f3ad-31a3-4b27-b383-8859526e81d4/ovnkube-controller/0.log"