Aug 03 23:27:45.830810 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 03 23:27:45.830824 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 03 23:27:45.830833 ip-10-0-136-126 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 03 23:27:45.831220 ip-10-0-136-126 systemd[1]: Failed to start Kubernetes Kubelet. Aug 03 23:27:55.514872 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 03 23:27:55.514891 ip-10-0-136-126 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c8fc1923fe1b4a7bb6f84a8547051c09 -- Aug 03 23:29:44.379271 ip-10-0-136-126 systemd[1]: Starting Kubernetes Kubelet... Aug 03 23:29:44.751602 ip-10-0-136-126 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 03 23:29:44.751602 ip-10-0-136-126 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 03 23:29:44.751602 ip-10-0-136-126 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 03 23:29:44.751602 ip-10-0-136-126 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 03 23:29:44.751602 ip-10-0-136-126 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 03 23:29:44.753625 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.753484 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 03 23:29:44.757861 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757842 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 03 23:29:44.757861 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757859 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 03 23:29:44.757861 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757863 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 03 23:29:44.757861 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757868 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757872 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757876 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757880 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757886 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757890 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757893 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757897 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757901 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757905 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757909 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757912 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757917 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757921 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757925 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757929 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757935 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757942 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757947 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 03 23:29:44.758115 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757951 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757956 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757960 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757965 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757988 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757993 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.757998 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758002 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758006 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758010 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758015 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758019 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758023 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758027 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758031 2556 feature_gate.go:328] unrecognized feature gate: Example2 Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758036 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758040 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758045 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758050 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758054 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 03 23:29:44.758885 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758058 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758062 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758066 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758071 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758074 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758078 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758083 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758087 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758092 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758096 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758100 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758106 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758110 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758114 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758118 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758122 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758126 2556 feature_gate.go:328] unrecognized feature gate: Example Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758130 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758134 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758139 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 03 23:29:44.759742 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758143 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758147 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758151 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758155 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758161 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758165 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758169 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758174 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758178 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758183 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758188 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758192 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758196 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758201 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758205 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758209 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758215 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758221 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758227 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 03 23:29:44.760608 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758232 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758237 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758242 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758246 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758251 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758255 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758261 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758268 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758273 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758278 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758282 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758286 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758290 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758294 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758298 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758302 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758306 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.758310 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759495 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 03 23:29:44.761445 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759505 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759510 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759514 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759519 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759524 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759528 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759532 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759536 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759540 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759544 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759549 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759553 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759560 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759568 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759573 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759578 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759585 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759590 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759595 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 03 23:29:44.762337 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759599 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759603 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759607 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759612 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759616 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759620 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759624 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759628 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759633 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759637 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759641 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759645 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759649 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759653 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759658 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759662 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759667 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759671 2556 feature_gate.go:328] unrecognized feature gate: Example Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759676 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759681 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 03 23:29:44.762902 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759685 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759689 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759693 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759697 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759701 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759706 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759710 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759715 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759720 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759724 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759728 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759732 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759736 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759740 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759744 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759748 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759751 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759756 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759761 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759765 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 03 23:29:44.763425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759769 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759773 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759777 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759781 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759786 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759790 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759795 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759800 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759804 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759809 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759813 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759828 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759833 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759837 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759841 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759849 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759853 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759858 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759862 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759866 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 03 23:29:44.763923 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759870 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759875 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759879 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759883 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759888 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759894 2556 feature_gate.go:328] unrecognized feature gate: Example2 Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759898 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759902 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759906 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759910 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759915 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759919 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759923 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759927 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759931 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759935 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759939 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759943 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.759950 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760541 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 03 23:29:44.764425 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760549 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760554 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760558 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760563 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760568 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760572 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760576 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760583 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760587 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760591 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760595 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760599 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760604 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760608 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760612 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760616 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760619 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760623 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760627 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760631 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 03 23:29:44.764922 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760635 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760639 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760643 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760648 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760652 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760656 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760660 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760664 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760669 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760672 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760678 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760682 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760686 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760690 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760694 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760698 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760703 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760707 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760711 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760717 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 03 23:29:44.765428 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760722 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760725 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760730 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760734 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760738 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760742 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760746 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760750 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760754 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760758 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760762 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760768 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760773 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760777 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760782 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760786 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760790 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760794 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760798 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 03 23:29:44.765916 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760804 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760810 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760816 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760822 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760827 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760831 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760838 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760843 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760848 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760853 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760858 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760863 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760869 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760874 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760878 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760882 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760886 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760890 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760894 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 03 23:29:44.766397 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760898 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760902 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760906 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760910 2556 feature_gate.go:328] unrecognized feature gate: Example Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760914 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760918 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760922 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760926 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760932 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760938 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760942 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760946 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760950 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760954 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760958 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760962 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760989 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760993 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.760997 2556 feature_gate.go:328] unrecognized feature gate: Example2 Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.761001 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 03 23:29:44.766844 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761093 2556 flags.go:64] FLAG: --address="0.0.0.0" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761104 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761113 2556 flags.go:64] FLAG: --anonymous-auth="true" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761120 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761128 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761136 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761142 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761148 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761153 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761159 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761165 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761171 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761176 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761181 2556 flags.go:64] FLAG: --cgroup-root="" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761185 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761190 2556 flags.go:64] FLAG: --client-ca-file="" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761194 2556 flags.go:64] FLAG: --cloud-provider="external" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761198 2556 flags.go:64] FLAG: --cluster-dns="[]" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761205 2556 flags.go:64] FLAG: --cluster-domain="" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761210 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761215 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761220 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761225 2556 flags.go:64] FLAG: --container-log-max-files="5" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761230 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 03 23:29:44.767378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761235 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761240 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761246 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761250 2556 flags.go:64] FLAG: --contention-profiling="false" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761257 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761262 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761267 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761271 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761278 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761282 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761287 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761291 2556 flags.go:64] FLAG: --enable-load-reader="false" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761296 2556 flags.go:64] FLAG: --enable-server="true" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761303 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761310 2556 flags.go:64] FLAG: --event-burst="100" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761315 2556 flags.go:64] FLAG: --event-qps="50" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761319 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761324 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761329 2556 flags.go:64] FLAG: --eviction-hard="" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761335 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761339 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761344 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761349 2556 flags.go:64] FLAG: --eviction-soft="" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761353 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761358 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 03 23:29:44.767956 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761363 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761368 2556 flags.go:64] FLAG: --experimental-mounter-path="" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761372 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761377 2556 flags.go:64] FLAG: --fail-swap-on="true" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761382 2556 flags.go:64] FLAG: --feature-gates="" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761388 2556 flags.go:64] FLAG: --file-check-frequency="20s" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761393 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761398 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761402 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761407 2556 flags.go:64] FLAG: --healthz-port="10248" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761411 2556 flags.go:64] FLAG: --help="false" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761418 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-136-126.ec2.internal" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761422 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761427 2556 flags.go:64] FLAG: --http-check-frequency="20s" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761432 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761437 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761442 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761446 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761450 2556 flags.go:64] FLAG: --image-service-endpoint="" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761454 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761460 2556 flags.go:64] FLAG: --kube-api-burst="100" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761465 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761471 2556 flags.go:64] FLAG: --kube-api-qps="50" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761475 2556 flags.go:64] FLAG: --kube-reserved="" Aug 03 23:29:44.768604 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761480 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761485 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761489 2556 flags.go:64] FLAG: --kubelet-cgroups="" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761494 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761499 2556 flags.go:64] FLAG: --lock-file="" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761503 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761508 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761512 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761521 2556 flags.go:64] FLAG: --log-json-split-stream="false" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761526 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761531 2556 flags.go:64] FLAG: --log-text-split-stream="false" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761535 2556 flags.go:64] FLAG: --logging-format="text" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761540 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761545 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761550 2556 flags.go:64] FLAG: --manifest-url="" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761554 2556 flags.go:64] FLAG: --manifest-url-header="" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761561 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761566 2556 flags.go:64] FLAG: --max-open-files="1000000" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761572 2556 flags.go:64] FLAG: --max-pods="110" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761577 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761582 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761586 2556 flags.go:64] FLAG: --memory-manager-policy="None" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761590 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761595 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761599 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 03 23:29:44.769205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761604 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761615 2556 flags.go:64] FLAG: --node-status-max-images="50" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761620 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761627 2556 flags.go:64] FLAG: --oom-score-adj="-999" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761632 2556 flags.go:64] FLAG: --pod-cidr="" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761637 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761647 2556 flags.go:64] FLAG: --pod-manifest-path="" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761651 2556 flags.go:64] FLAG: --pod-max-pids="-1" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761656 2556 flags.go:64] FLAG: --pods-per-core="0" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761661 2556 flags.go:64] FLAG: --port="10250" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761665 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761670 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a15de0860657623" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761675 2556 flags.go:64] FLAG: --qos-reserved="" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761679 2556 flags.go:64] FLAG: --read-only-port="10255" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761684 2556 flags.go:64] FLAG: --register-node="true" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761689 2556 flags.go:64] FLAG: --register-with-taints="" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761695 2556 flags.go:64] FLAG: --registry-burst="10" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761700 2556 flags.go:64] FLAG: --registry-qps="5" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761704 2556 flags.go:64] FLAG: --reserved-cpus="" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761708 2556 flags.go:64] FLAG: --reserved-memory="" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761714 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761719 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761724 2556 flags.go:64] FLAG: --rotate-certificates="false" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761728 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761733 2556 flags.go:64] FLAG: --runonce="false" Aug 03 23:29:44.769801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761738 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761743 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761748 2556 flags.go:64] FLAG: --seccomp-default="false" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761753 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761757 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761762 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761767 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761772 2556 flags.go:64] FLAG: --storage-driver-password="root" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761777 2556 flags.go:64] FLAG: --storage-driver-secure="false" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761781 2556 flags.go:64] FLAG: --storage-driver-table="stats" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761788 2556 flags.go:64] FLAG: --storage-driver-user="root" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761793 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761797 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761803 2556 flags.go:64] FLAG: --system-cgroups="" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761807 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761817 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761821 2556 flags.go:64] FLAG: --tls-cert-file="" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761827 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761832 2556 flags.go:64] FLAG: --tls-min-version="" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761836 2556 flags.go:64] FLAG: --tls-private-key-file="" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761841 2556 flags.go:64] FLAG: --topology-manager-policy="none" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761846 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761850 2556 flags.go:64] FLAG: --topology-manager-scope="container" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761855 2556 flags.go:64] FLAG: --v="2" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761861 2556 flags.go:64] FLAG: --version="false" Aug 03 23:29:44.770423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761867 2556 flags.go:64] FLAG: --vmodule="" Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761873 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.761878 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762083 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762092 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762097 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762102 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762107 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762111 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762116 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762121 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762125 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762130 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762135 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762140 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762144 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762148 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762155 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762159 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762163 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 03 23:29:44.771255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762167 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762172 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762176 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762181 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762185 2556 feature_gate.go:328] unrecognized feature gate: Example2 Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762190 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762194 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762198 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762202 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762207 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762210 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762214 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762219 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762223 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762227 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762232 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762236 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762240 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762244 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762248 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 03 23:29:44.771952 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762252 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762257 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762261 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762265 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762269 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762274 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762278 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762282 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762286 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762292 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762296 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762300 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762304 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762309 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762314 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762318 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762322 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762327 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762331 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762334 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 03 23:29:44.772497 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762338 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762342 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762346 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762351 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762354 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762358 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762362 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762366 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762370 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762375 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762379 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762383 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762387 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762393 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762399 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762405 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762409 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762413 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762418 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 03 23:29:44.773139 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762422 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762427 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762432 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762436 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762441 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762445 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762449 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762453 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762460 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762466 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762472 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762478 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762483 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762488 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762492 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762497 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762502 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762507 2556 feature_gate.go:328] unrecognized feature gate: Example Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762512 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 03 23:29:44.773596 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762516 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762520 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762525 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.762560 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.762571 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.769339 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.769352 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.770637 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.770644 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770700 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770706 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770709 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770713 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770716 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770719 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 03 23:29:44.774069 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770722 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770725 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770727 2556 feature_gate.go:328] unrecognized feature gate: Example Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770730 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770733 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770736 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770738 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770741 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770743 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770746 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770748 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770751 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770754 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770756 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770759 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770761 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770764 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770767 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770770 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770772 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 03 23:29:44.774439 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770807 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770846 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770851 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770855 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770858 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770861 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770864 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770866 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770869 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770872 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770875 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770878 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770880 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770883 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770889 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770892 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770894 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770897 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770900 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 03 23:29:44.774908 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770903 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770905 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.770908 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.771914 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772008 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772015 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772021 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772025 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772030 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772035 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772039 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772043 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772046 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772049 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772052 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772059 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772214 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772222 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772225 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 03 23:29:44.775407 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772229 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772231 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772234 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772237 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772239 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772243 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772245 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772248 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772252 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772256 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772260 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772264 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772269 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772274 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772277 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772281 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772286 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772289 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772292 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772295 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 03 23:29:44.775886 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772298 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772300 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772303 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772305 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772308 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772310 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772313 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772315 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772318 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772321 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772323 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772326 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772329 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772331 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772334 2556 feature_gate.go:328] unrecognized feature gate: Example2 Aug 03 23:29:44.776404 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.772340 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772463 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772467 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772470 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772473 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772475 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772478 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772481 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772483 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772486 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772488 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772491 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772494 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772497 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772500 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772502 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772505 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772508 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772511 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772514 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 03 23:29:44.776777 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772516 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772519 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772521 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772524 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772526 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772529 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772531 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772534 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772536 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772539 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772541 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772543 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772546 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772549 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772551 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772553 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772556 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772559 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772561 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772564 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 03 23:29:44.777348 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772566 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772569 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772571 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772574 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772577 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772579 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772582 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772584 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772588 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772591 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772594 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772596 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772599 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772601 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772604 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772606 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772609 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772611 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772614 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 03 23:29:44.777826 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772616 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772619 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772623 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772626 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772629 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772632 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772634 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772637 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772640 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772643 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772646 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772648 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772651 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772653 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772656 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772658 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772661 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772664 2556 feature_gate.go:328] unrecognized feature gate: Example2 Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772667 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 03 23:29:44.778306 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772669 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772673 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772675 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772678 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772680 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772683 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772685 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772688 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772692 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772695 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772698 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772702 2556 feature_gate.go:328] unrecognized feature gate: Example Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772704 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772707 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772710 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772712 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772715 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772717 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772720 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772722 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 03 23:29:44.778786 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772725 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 03 23:29:44.779305 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:44.772727 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 03 23:29:44.779305 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.772732 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 03 23:29:44.779305 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.772829 2556 server.go:962] "Client rotation is on, will bootstrap in background" Aug 03 23:29:44.779305 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.775064 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 03 23:29:44.779305 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.776483 2556 server.go:1019] "Starting client certificate rotation" Aug 03 23:29:44.779305 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.776585 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 03 23:29:44.779305 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.776617 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 03 23:29:44.779305 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.779136 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 03 23:29:44.782204 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.782183 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 03 23:29:44.790811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.790797 2556 log.go:25] "Validated CRI v1 runtime API" Aug 03 23:29:44.793951 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.793938 2556 log.go:25] "Validated CRI v1 image API" Aug 03 23:29:44.795569 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.795554 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 03 23:29:44.798566 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.798548 2556 fs.go:135] Filesystem UUIDs: map[39293774-078d-4d65-b9bd-410e4a185f41:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e5db8041-2d3a-45e9-b37e-37c069895bcf:/dev/nvme0n1p4] Aug 03 23:29:44.798634 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.798565 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 03 23:29:44.804569 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.804471 2556 manager.go:216] Machine: {Timestamp:2026-08-03 23:29:44.803262734 +0000 UTC m=+0.312853464 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3280343 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cd0c6e45eaf3f0aaa043bb91aa65c SystemUUID:ec2cd0c6-e45e-af3f-0aaa-043bb91aa65c BootID:c8fc1923-fe1b-4a7b-b6f8-4a8547051c09 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0b:fc:60:85:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0b:fc:60:85:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:ca:d8:91:29:ae Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 03 23:29:44.804569 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.804565 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 03 23:29:44.804670 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.804641 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 03 23:29:44.806229 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.806215 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 03 23:29:44.807777 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.807751 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 03 23:29:44.807908 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.807782 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-126.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 03 23:29:44.808561 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.808550 2556 topology_manager.go:138] "Creating topology manager with none policy" Aug 03 23:29:44.808598 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.808564 2556 container_manager_linux.go:309] "Creating device plugin manager" Aug 03 23:29:44.808598 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.808572 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 03 23:29:44.809703 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.809693 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 03 23:29:44.810825 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.810815 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 03 23:29:44.814880 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.814869 2556 state_mem.go:36] "Initialized new in-memory state store" Aug 03 23:29:44.815019 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.815010 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 03 23:29:44.815058 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.815045 2556 kubelet.go:486] "Attempting to sync node with API server" Aug 03 23:29:44.815058 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.815055 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 03 23:29:44.815119 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.815072 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 03 23:29:44.815119 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.815088 2556 kubelet.go:398] "Adding apiserver pod source" Aug 03 23:29:44.815119 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.815099 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 03 23:29:44.815664 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.815649 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 03 23:29:44.815714 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.815677 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 03 23:29:44.819827 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.819810 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 03 23:29:44.821644 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.821630 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 03 23:29:44.821691 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.821662 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 03 23:29:44.823038 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823025 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823041 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823048 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823053 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823062 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823067 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823076 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823091 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823102 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 03 23:29:44.823109 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823107 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 03 23:29:44.823346 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823116 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 03 23:29:44.823346 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823122 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 03 23:29:44.823806 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823796 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 03 23:29:44.823836 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.823808 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 03 23:29:44.827438 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.827424 2556 server.go:1268] "Started kubelet" Aug 03 23:29:44.827533 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.827495 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 03 23:29:44.827607 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.827566 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 03 23:29:44.827655 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.827629 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 03 23:29:44.828015 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.827995 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-126.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 03 23:29:44.828132 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.828113 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 03 23:29:44.828182 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.828161 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 03 23:29:44.828342 ip-10-0-136-126 systemd[1]: Started Kubernetes Kubelet. Aug 03 23:29:44.829485 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.829456 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 03 23:29:44.830000 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.829965 2556 server.go:310] "Adding debug handlers to kubelet server" Aug 03 23:29:44.834474 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.834446 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 03 23:29:44.834474 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.833320 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-126.ec2.internal.18c86f71569b019d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-126.ec2.internal,UID:ip-10-0-136-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-126.ec2.internal,},FirstTimestamp:2026-08-03 23:29:44.827388317 +0000 UTC m=+0.336979048,LastTimestamp:2026-08-03 23:29:44.827388317 +0000 UTC m=+0.336979048,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-126.ec2.internal,}" Aug 03 23:29:44.835438 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.835424 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 03 23:29:44.835510 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.835425 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 03 23:29:44.835566 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.835555 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:44.836317 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.836169 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 03 23:29:44.836317 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.836200 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 03 23:29:44.836317 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.836169 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 03 23:29:44.836535 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.836324 2556 reconstruct.go:97] "Volume reconstruction finished" Aug 03 23:29:44.836535 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.836334 2556 reconciler.go:29] "Reconciler: start to sync state" Aug 03 23:29:44.836869 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.836848 2556 factory.go:55] Registering systemd factory Aug 03 23:29:44.836952 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.836872 2556 factory.go:223] Registration of the systemd container factory successfully Aug 03 23:29:44.837142 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.837128 2556 factory.go:153] Registering CRI-O factory Aug 03 23:29:44.837207 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.837146 2556 factory.go:223] Registration of the crio container factory successfully Aug 03 23:29:44.837207 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.837194 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 03 23:29:44.837291 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.837218 2556 factory.go:103] Registering Raw factory Aug 03 23:29:44.837291 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.837232 2556 manager.go:1196] Started watching for new ooms in manager Aug 03 23:29:44.837856 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.837841 2556 manager.go:349] Starting recovery of all containers Aug 03 23:29:44.840627 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.840608 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 03 23:29:44.840820 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.840797 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 03 23:29:44.848939 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.848919 2556 manager.go:354] Recovery completed Aug 03 23:29:44.853293 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.853279 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 03 23:29:44.856299 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.856271 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Aug 03 23:29:44.856368 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.856310 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Aug 03 23:29:44.856368 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.856321 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Aug 03 23:29:44.856902 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.856889 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 03 23:29:44.856939 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.856903 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 03 23:29:44.856939 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.856918 2556 state_mem.go:36] "Initialized new in-memory state store" Aug 03 23:29:44.857024 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.856964 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 03 23:29:44.857722 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.857706 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Aug 03 23:29:44.857774 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.857736 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Aug 03 23:29:44.857774 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.857748 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Aug 03 23:29:44.858284 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.858273 2556 policy_none.go:49] "None policy: Start" Aug 03 23:29:44.858364 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.858288 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 03 23:29:44.858364 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.858298 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 03 23:29:44.858489 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.858398 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-126.ec2.internal.18c86f7158541d2c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-126.ec2.internal,UID:ip-10-0-136-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-126.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-126.ec2.internal,},FirstTimestamp:2026-08-03 23:29:44.856296748 +0000 UTC m=+0.365887477,LastTimestamp:2026-08-03 23:29:44.856296748 +0000 UTC m=+0.365887477,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-126.ec2.internal,}" Aug 03 23:29:44.859319 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.859308 2556 policy_none.go:47] "Start" Aug 03 23:29:44.860264 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.860156 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zmkvp" Aug 03 23:29:44.868606 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.868534 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-126.ec2.internal.18c86f7158546309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-126.ec2.internal,UID:ip-10-0-136-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-126.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-126.ec2.internal,},FirstTimestamp:2026-08-03 23:29:44.856314633 +0000 UTC m=+0.365905362,LastTimestamp:2026-08-03 23:29:44.856314633 +0000 UTC m=+0.365905362,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-126.ec2.internal,}" Aug 03 23:29:44.870051 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.870035 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zmkvp" Aug 03 23:29:44.889518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.889502 2556 manager.go:337] "Starting Device Plugin manager" Aug 03 23:29:44.889593 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.889553 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 03 23:29:44.889593 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.889565 2556 server.go:87] "Starting device plugin registration server" Aug 03 23:29:44.889800 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.889788 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 03 23:29:44.889855 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.889802 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 03 23:29:44.889934 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.889917 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 03 23:29:44.890057 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.890044 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 03 23:29:44.890057 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.890057 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 03 23:29:44.890600 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.890579 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 03 23:29:44.890680 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.890631 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:44.966793 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.966753 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 03 23:29:44.968063 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.968043 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 03 23:29:44.968158 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.968083 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 03 23:29:44.968158 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.968106 2556 kubelet.go:2448] "Starting kubelet main sync loop" Aug 03 23:29:44.968158 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:44.968145 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 03 23:29:44.970266 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.970250 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 03 23:29:44.990271 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.990255 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 03 23:29:44.991095 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.991072 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Aug 03 23:29:44.991169 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.991112 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Aug 03 23:29:44.991169 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.991125 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Aug 03 23:29:44.991169 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:44.991147 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.000289 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.000271 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.000367 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.000290 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-126.ec2.internal\": node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:45.029551 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.029508 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:45.069160 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.069138 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal"] Aug 03 23:29:45.069219 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.069204 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 03 23:29:45.070044 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.070030 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Aug 03 23:29:45.070122 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.070053 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Aug 03 23:29:45.070122 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.070062 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Aug 03 23:29:45.071174 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071161 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 03 23:29:45.071304 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071289 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.071359 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071325 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 03 23:29:45.071867 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071852 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Aug 03 23:29:45.071917 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071859 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Aug 03 23:29:45.071917 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071888 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Aug 03 23:29:45.071917 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071901 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Aug 03 23:29:45.072077 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071874 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Aug 03 23:29:45.072077 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.071957 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Aug 03 23:29:45.073190 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.073175 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.073271 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.073206 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 03 23:29:45.073869 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.073856 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Aug 03 23:29:45.073925 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.073876 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Aug 03 23:29:45.073925 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.073887 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Aug 03 23:29:45.099956 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.099936 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-126.ec2.internal\" not found" node="ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.103987 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.103955 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-126.ec2.internal\" not found" node="ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.130350 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.130333 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:45.138444 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.138428 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8b7e0082a8d432502c091a2ad7a9c78-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"d8b7e0082a8d432502c091a2ad7a9c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.138506 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.138452 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eba9639115ca8e3bac669ba1dab6cd0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"eba9639115ca8e3bac669ba1dab6cd0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.138506 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.138470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba9639115ca8e3bac669ba1dab6cd0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"eba9639115ca8e3bac669ba1dab6cd0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.231226 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.231180 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:45.239521 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.239497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8b7e0082a8d432502c091a2ad7a9c78-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"d8b7e0082a8d432502c091a2ad7a9c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.239581 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.239526 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eba9639115ca8e3bac669ba1dab6cd0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"eba9639115ca8e3bac669ba1dab6cd0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.239581 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.239543 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba9639115ca8e3bac669ba1dab6cd0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"eba9639115ca8e3bac669ba1dab6cd0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.239654 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.239578 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba9639115ca8e3bac669ba1dab6cd0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"eba9639115ca8e3bac669ba1dab6cd0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.239654 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.239584 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d8b7e0082a8d432502c091a2ad7a9c78-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"d8b7e0082a8d432502c091a2ad7a9c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.239654 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.239609 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eba9639115ca8e3bac669ba1dab6cd0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"eba9639115ca8e3bac669ba1dab6cd0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.331943 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.331882 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:45.403445 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.403419 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.408560 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:45.408505 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:45.432850 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.432831 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:45.533412 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.533385 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:45.633909 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:45.633850 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:47.180879 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:47.180858 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:47.222294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.222274 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 03 23:29:47.222667 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.222419 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 03 23:29:47.281427 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:47.281397 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:47.321137 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.321099 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-02 23:24:46 +0000 UTC" deadline="2028-02-07 06:40:03.862602891 +0000 UTC" Aug 03 23:29:47.321137 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.321135 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13255h10m16.54147099s" Aug 03 23:29:47.382446 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:47.382389 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:47.384547 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.384529 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 03 23:29:47.385862 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.385840 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Aug 03 23:29:47.385940 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.385877 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Aug 03 23:29:47.385940 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.385892 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Aug 03 23:29:47.390053 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.390036 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 03 23:29:47.404255 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:47.404228 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b7e0082a8d432502c091a2ad7a9c78.slice/crio-bf62151efab142548cdd719e0bf5bc1e19ae52a645d47240bc7b735a35e59987 WatchSource:0}: Error finding container bf62151efab142548cdd719e0bf5bc1e19ae52a645d47240bc7b735a35e59987: Status 404 returned error can't find the container with id bf62151efab142548cdd719e0bf5bc1e19ae52a645d47240bc7b735a35e59987 Aug 03 23:29:47.404661 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:47.404642 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba9639115ca8e3bac669ba1dab6cd0e.slice/crio-0ef5a1c74f0e95057541dde6d083018315b9a58a726abd9e5c144c90a3f5e77a WatchSource:0}: Error finding container 0ef5a1c74f0e95057541dde6d083018315b9a58a726abd9e5c144c90a3f5e77a: Status 404 returned error can't find the container with id 0ef5a1c74f0e95057541dde6d083018315b9a58a726abd9e5c144c90a3f5e77a Aug 03 23:29:47.409723 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.409709 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 03 23:29:47.418241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.418202 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"eba9639115ca8e3bac669ba1dab6cd0e","Type":"ContainerStarted","Data":"0ef5a1c74f0e95057541dde6d083018315b9a58a726abd9e5c144c90a3f5e77a"} Aug 03 23:29:47.419168 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.419149 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" event={"ID":"d8b7e0082a8d432502c091a2ad7a9c78","Type":"ContainerStarted","Data":"bf62151efab142548cdd719e0bf5bc1e19ae52a645d47240bc7b735a35e59987"} Aug 03 23:29:47.482938 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:47.482908 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:47.578338 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.578301 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 03 23:29:47.583385 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:47.583368 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:47.683844 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:47.683779 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:47.784558 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:47.784527 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Aug 03 23:29:47.831292 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.831263 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 03 23:29:47.882023 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.881994 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Aug 03 23:29:47.895354 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.895333 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 03 23:29:47.896561 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.896549 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Aug 03 23:29:47.904543 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:47.904531 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 03 23:29:48.208571 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.208536 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 03 23:29:48.263037 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.262747 2556 apiserver.go:52] "Watching apiserver" Aug 03 23:29:48.273326 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.273287 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 03 23:29:48.273865 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.273829 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-4zqgb","kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal","openshift-dns/node-resolver-7r62k","openshift-image-registry/node-ca-nsl2l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal","openshift-multus/network-metrics-daemon-jsxr8","openshift-network-diagnostics/network-check-target-t7st8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb","openshift-cluster-node-tuning-operator/tuned-6zhpg","openshift-multus/multus-7tb9z","openshift-multus/multus-additional-cni-plugins-dxfpf","openshift-network-operator/iptables-alerter-hzv6w","openshift-ovn-kubernetes/ovnkube-node-g9mb6","kube-system/global-pull-secret-syncer-nd4xg"] Aug 03 23:29:48.275291 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.275256 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:48.275409 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.275342 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:29:48.276554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.276524 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.277605 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.277584 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.278982 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.278964 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.279128 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.279110 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-52d26\"" Aug 03 23:29:48.279632 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.279615 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 03 23:29:48.279784 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.279761 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 03 23:29:48.279876 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.279824 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 03 23:29:48.279950 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.279770 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 03 23:29:48.280003 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.279750 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 03 23:29:48.280073 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.279921 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 03 23:29:48.280155 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.280138 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hhcnt\"" Aug 03 23:29:48.280240 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.280216 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 03 23:29:48.280868 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.280850 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.280961 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.280920 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:29:48.281261 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.281242 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 03 23:29:48.281452 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.281432 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 03 23:29:48.281520 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.281485 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 03 23:29:48.281575 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.281543 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hbdz7\"" Aug 03 23:29:48.282217 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.282195 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:48.282336 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.282266 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:29:48.282912 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.282894 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 03 23:29:48.283383 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.283364 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.284824 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.284804 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.285633 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.285614 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 03 23:29:48.285734 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.285712 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 03 23:29:48.285888 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.285870 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r62jc\"" Aug 03 23:29:48.287146 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.287127 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 03 23:29:48.287264 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.287159 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nnkp8\"" Aug 03 23:29:48.287383 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.287362 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 03 23:29:48.287502 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.287482 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 03 23:29:48.288561 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.288540 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 03 23:29:48.288781 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.288762 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 03 23:29:48.288988 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.288972 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 03 23:29:48.293778 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.293692 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.294400 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294378 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:48.294488 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294423 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.294488 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294457 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-cni-bin\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.294488 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294484 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-kubernetes\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.294677 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294509 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-lib-modules\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.294677 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294539 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-run-netns\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.294677 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294567 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovn-node-metrics-cert\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.294677 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294595 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6bf\" (UniqueName: \"kubernetes.io/projected/30f44684-cb92-47de-9a0f-ffc6ca6346e6-kube-api-access-hg6bf\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.294677 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294624 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-run\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.294677 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294646 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294673 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36bfdb11-2080-4407-9208-2d5ef6995e91-dbus\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294712 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysconfig\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294740 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-tuned\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294766 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-tmp\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294790 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294817 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30f44684-cb92-47de-9a0f-ffc6ca6346e6-host\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-var-lib-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294878 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.294930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-system-cni-dir\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294938 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.294966 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295001 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdl4m\" (UniqueName: \"kubernetes.io/projected/29374835-d52e-403b-9834-071b2f9bda81-kube-api-access-xdl4m\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295027 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/347d49d7-131d-4942-82ea-6ce44f73495d-tmp-dir\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295053 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-sys\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295079 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295103 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-modprobe-d\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-systemd\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295160 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-env-overrides\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295188 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbbh\" (UniqueName: \"kubernetes.io/projected/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-kube-api-access-pjbbh\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295210 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-os-release\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295239 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cj8\" (UniqueName: \"kubernetes.io/projected/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-kube-api-access-z9cj8\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295266 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-log-socket\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.295297 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295293 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6wx\" (UniqueName: \"kubernetes.io/projected/347d49d7-131d-4942-82ea-6ce44f73495d-kube-api-access-ml6wx\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.295898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295339 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysctl-conf\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.295898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-var-lib-kubelet\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.295898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295397 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cnibin\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.295898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295420 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-systemd-units\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.295898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295448 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:48.295898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295481 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-ovn\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.295898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295509 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-cni-netd\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.295898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295536 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30f44684-cb92-47de-9a0f-ffc6ca6346e6-serviceca\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.296335 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295907 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-kubelet\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.296335 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.295967 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-slash\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.296335 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-systemd\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.296335 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296092 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-node-log\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.296335 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296125 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovnkube-config\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.296559 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296437 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 03 23:29:48.296592 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296581 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2fgvc\"" Aug 03 23:29:48.296689 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296672 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 03 23:29:48.296941 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296919 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovnkube-script-lib\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.297019 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296965 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.297019 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.296996 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/347d49d7-131d-4942-82ea-6ce44f73495d-hosts-file\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.297019 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.297000 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 03 23:29:48.297172 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.297062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqkq\" (UniqueName: \"kubernetes.io/projected/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-kube-api-access-qqqkq\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.297172 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.297097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysctl-d\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.297172 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.297121 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-host\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.297334 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.297152 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36bfdb11-2080-4407-9208-2d5ef6995e91-kubelet-config\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.297334 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.297231 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-etc-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.297703 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.297684 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.299103 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.299087 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:29:48.300864 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.300380 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 03 23:29:48.300864 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.300606 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jv85z\"" Aug 03 23:29:48.300864 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.300837 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.302576 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.301823 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 03 23:29:48.302576 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.302015 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 03 23:29:48.302576 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.302379 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cwqtl\"" Aug 03 23:29:48.303179 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.303161 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 03 23:29:48.303255 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.303234 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 03 23:29:48.303395 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.303376 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 03 23:29:48.303506 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.303438 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5jrxl\"" Aug 03 23:29:48.304798 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.304780 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 03 23:29:48.322023 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.322003 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8q95x" Aug 03 23:29:48.328499 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.328482 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8q95x" Aug 03 23:29:48.384096 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.384070 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 03 23:29:48.398183 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398155 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cj8\" (UniqueName: \"kubernetes.io/projected/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-kube-api-access-z9cj8\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.398294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398192 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-log-socket\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398218 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6wx\" (UniqueName: \"kubernetes.io/projected/347d49d7-131d-4942-82ea-6ce44f73495d-kube-api-access-ml6wx\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.398294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398244 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-cni-bin\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.398294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398270 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysctl-conf\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.398518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398293 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-var-lib-kubelet\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.398518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398336 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95fc751e-3c56-4f22-97c9-e0664b53b97f-cni-binary-copy\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.398518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398367 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-etc-kubernetes\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.398518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398391 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cnibin\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.398518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398416 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-systemd-units\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398440 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f26x6\" (UniqueName: \"kubernetes.io/projected/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-kube-api-access-f26x6\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.398518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398467 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-socket-dir-parent\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.398518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398504 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-cni-multus\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398537 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-kubelet\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398567 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-ovn\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398630 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-cni-netd\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398654 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-device-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398678 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8903faa-2d47-47a2-a7c0-ef8d5f7fb148-agent-certs\") pod \"konnectivity-agent-4zqgb\" (UID: \"d8903faa-2d47-47a2-a7c0-ef8d5f7fb148\") " pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398707 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10e2b910-b064-4774-91b6-b9f215f60e33-host-slash\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398725 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysctl-conf\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398731 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30f44684-cb92-47de-9a0f-ffc6ca6346e6-serviceca\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398778 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-kubelet\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398802 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-slash\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398830 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-systemd\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398855 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-node-log\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398878 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovnkube-config\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.398897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398902 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovnkube-script-lib\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398931 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-sys-fs\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398981 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/347d49d7-131d-4942-82ea-6ce44f73495d-hosts-file\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399004 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-socket-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399030 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399058 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10e2b910-b064-4774-91b6-b9f215f60e33-iptables-alerter-script\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399084 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-os-release\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-daemon-config\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399137 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-multus-certs\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399167 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30f44684-cb92-47de-9a0f-ffc6ca6346e6-serviceca\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399174 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqkq\" (UniqueName: \"kubernetes.io/projected/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-kube-api-access-qqqkq\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399213 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysctl-d\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399237 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-host\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399261 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399287 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cnibin\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399347 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-kubelet\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.399612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399359 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-systemd-units\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399388 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-slash\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.398538 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-var-lib-kubelet\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399429 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-systemd\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399447 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-log-socket\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399485 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-node-log\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.399536 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.399597 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:29:48.89957234 +0000 UTC m=+2.962828751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399928 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-ovn\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399957 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovnkube-config\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399980 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-cni-netd\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.399285 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-system-cni-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-netns\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400073 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-conf-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400099 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36bfdb11-2080-4407-9208-2d5ef6995e91-kubelet-config\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-etc-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400148 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-cnibin\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.400429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400174 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400201 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400228 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-cni-bin\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400253 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8903faa-2d47-47a2-a7c0-ef8d5f7fb148-konnectivity-ca\") pod \"konnectivity-agent-4zqgb\" (UID: \"d8903faa-2d47-47a2-a7c0-ef8d5f7fb148\") " pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400278 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqqh5\" (UniqueName: \"kubernetes.io/projected/95fc751e-3c56-4f22-97c9-e0664b53b97f-kube-api-access-kqqh5\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400323 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-kubernetes\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400351 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-lib-modules\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400374 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-run-netns\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovn-node-metrics-cert\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400424 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv47n\" (UniqueName: \"kubernetes.io/projected/10e2b910-b064-4774-91b6-b9f215f60e33-kube-api-access-fv47n\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400431 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/36bfdb11-2080-4407-9208-2d5ef6995e91-kubelet-config\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400448 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6bf\" (UniqueName: \"kubernetes.io/projected/30f44684-cb92-47de-9a0f-ffc6ca6346e6-kube-api-access-hg6bf\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400461 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-etc-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400471 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-run\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400489 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-kubernetes\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400514 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-cni-bin\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400559 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36bfdb11-2080-4407-9208-2d5ef6995e91-dbus\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400584 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysconfig\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400586 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/347d49d7-131d-4942-82ea-6ce44f73495d-hosts-file\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400612 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-tuned\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-tmp\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400642 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-lib-modules\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400643 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400669 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400678 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30f44684-cb92-47de-9a0f-ffc6ca6346e6-host\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400702 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysconfig\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400721 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-var-lib-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400933 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.400952 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovnkube-script-lib\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401013 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-var-lib-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30f44684-cb92-47de-9a0f-ffc6ca6346e6-host\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401231 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401264 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-cni-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401286 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-hostroot\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.401811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401335 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-system-cni-dir\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401359 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401423 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401448 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdl4m\" (UniqueName: \"kubernetes.io/projected/29374835-d52e-403b-9834-071b2f9bda81-kube-api-access-xdl4m\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401469 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/347d49d7-131d-4942-82ea-6ce44f73495d-tmp-dir\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401488 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-sys\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401529 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-registration-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401552 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-k8s-cni-cncf-io\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-modprobe-d\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401596 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-systemd\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401606 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-run\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401196 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-sysctl-d\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401656 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-env-overrides\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401678 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbbh\" (UniqueName: \"kubernetes.io/projected/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-kube-api-access-pjbbh\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401686 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-system-cni-dir\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.402463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401699 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-os-release\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401718 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-systemd\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401789 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/347d49d7-131d-4942-82ea-6ce44f73495d-tmp-dir\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401815 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401796 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-os-release\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401854 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-run-openvswitch\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401878 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/36bfdb11-2080-4407-9208-2d5ef6995e91-dbus\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401883 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-run-ovn-kubernetes\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401832 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-sys\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-host-run-netns\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.401980 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-modprobe-d\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.402010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-host\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.402070 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.402146 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret podName:36bfdb11-2080-4407-9208-2d5ef6995e91 nodeName:}" failed. No retries permitted until 2026-08-03 23:29:48.902130055 +0000 UTC m=+2.965386477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret") pod "global-pull-secret-syncer-nd4xg" (UID: "36bfdb11-2080-4407-9208-2d5ef6995e91") : object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.402501 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-env-overrides\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.403241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.402909 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.404003 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.403249 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.404702 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.404679 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-etc-tuned\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.405012 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.404983 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-ovn-node-metrics-cert\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.406620 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.406589 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-tmp\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.413711 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.413690 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 03 23:29:48.413807 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.413714 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 03 23:29:48.413807 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.413729 2556 projected.go:196] Error preparing data for projected volume kube-api-access-2c84m for pod openshift-network-diagnostics/network-check-target-t7st8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:48.413807 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.413783 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m podName:57270cca-6d1b-4419-a389-ecdd66c5c92a nodeName:}" failed. No retries permitted until 2026-08-03 23:29:48.91376908 +0000 UTC m=+2.977025509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2c84m" (UniqueName: "kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m") pod "network-check-target-t7st8" (UID: "57270cca-6d1b-4419-a389-ecdd66c5c92a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:48.416268 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.416244 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cj8\" (UniqueName: \"kubernetes.io/projected/4aaa22d4-2499-45d2-9a81-1c2aaa7cce02-kube-api-access-z9cj8\") pod \"tuned-6zhpg\" (UID: \"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02\") " pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.416708 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.416687 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6bf\" (UniqueName: \"kubernetes.io/projected/30f44684-cb92-47de-9a0f-ffc6ca6346e6-kube-api-access-hg6bf\") pod \"node-ca-nsl2l\" (UID: \"30f44684-cb92-47de-9a0f-ffc6ca6346e6\") " pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.417380 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.417164 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdl4m\" (UniqueName: \"kubernetes.io/projected/29374835-d52e-403b-9834-071b2f9bda81-kube-api-access-xdl4m\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:48.420418 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.420208 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbbh\" (UniqueName: \"kubernetes.io/projected/f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e-kube-api-access-pjbbh\") pod \"ovnkube-node-g9mb6\" (UID: \"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e\") " pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.420963 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.420908 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqkq\" (UniqueName: \"kubernetes.io/projected/73bcb60f-5b25-4b51-8d80-4bc2f5276f12-kube-api-access-qqqkq\") pod \"multus-additional-cni-plugins-dxfpf\" (UID: \"73bcb60f-5b25-4b51-8d80-4bc2f5276f12\") " pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.423460 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.423433 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6wx\" (UniqueName: \"kubernetes.io/projected/347d49d7-131d-4942-82ea-6ce44f73495d-kube-api-access-ml6wx\") pod \"node-resolver-7r62k\" (UID: \"347d49d7-131d-4942-82ea-6ce44f73495d\") " pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.502540 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502456 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-cni-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502540 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-hostroot\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502540 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502525 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-registration-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502549 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-k8s-cni-cncf-io\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502579 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-cni-bin\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502587 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-cni-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502604 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95fc751e-3c56-4f22-97c9-e0664b53b97f-cni-binary-copy\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502619 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-registration-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502631 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-k8s-cni-cncf-io\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-etc-kubernetes\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502590 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-hostroot\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502679 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-etc-kubernetes\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502689 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f26x6\" (UniqueName: \"kubernetes.io/projected/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-kube-api-access-f26x6\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-cni-bin\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.502766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-socket-dir-parent\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-cni-multus\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502812 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-kubelet\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502850 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-device-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8903faa-2d47-47a2-a7c0-ef8d5f7fb148-agent-certs\") pod \"konnectivity-agent-4zqgb\" (UID: \"d8903faa-2d47-47a2-a7c0-ef8d5f7fb148\") " pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10e2b910-b064-4774-91b6-b9f215f60e33-host-slash\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502928 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-sys-fs\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502952 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-socket-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502970 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-socket-dir-parent\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.502975 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10e2b910-b064-4774-91b6-b9f215f60e33-iptables-alerter-script\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503024 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-os-release\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503051 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-daemon-config\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503073 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-multus-certs\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503078 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503095 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503111 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-system-cni-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.503355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503115 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-cni-multus\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-netns\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503150 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-conf-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503158 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-var-lib-kubelet\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503170 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-cnibin\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503170 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95fc751e-3c56-4f22-97c9-e0664b53b97f-cni-binary-copy\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503196 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8903faa-2d47-47a2-a7c0-ef8d5f7fb148-konnectivity-ca\") pod \"konnectivity-agent-4zqgb\" (UID: \"d8903faa-2d47-47a2-a7c0-ef8d5f7fb148\") " pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503208 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-device-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503230 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10e2b910-b064-4774-91b6-b9f215f60e33-host-slash\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503245 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqqh5\" (UniqueName: \"kubernetes.io/projected/95fc751e-3c56-4f22-97c9-e0664b53b97f-kube-api-access-kqqh5\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-sys-fs\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv47n\" (UniqueName: \"kubernetes.io/projected/10e2b910-b064-4774-91b6-b9f215f60e33-kube-api-access-fv47n\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503388 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-socket-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503439 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503525 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-system-cni-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503558 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-netns\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503585 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-conf-dir\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503621 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-cnibin\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503903 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95fc751e-3c56-4f22-97c9-e0664b53b97f-multus-daemon-config\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504800 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503957 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-os-release\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504800 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.503991 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10e2b910-b064-4774-91b6-b9f215f60e33-iptables-alerter-script\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.504800 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.504017 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95fc751e-3c56-4f22-97c9-e0664b53b97f-host-run-multus-certs\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.504800 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.504133 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d8903faa-2d47-47a2-a7c0-ef8d5f7fb148-konnectivity-ca\") pod \"konnectivity-agent-4zqgb\" (UID: \"d8903faa-2d47-47a2-a7c0-ef8d5f7fb148\") " pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:29:48.505844 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.505816 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d8903faa-2d47-47a2-a7c0-ef8d5f7fb148-agent-certs\") pod \"konnectivity-agent-4zqgb\" (UID: \"d8903faa-2d47-47a2-a7c0-ef8d5f7fb148\") " pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:29:48.516946 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.516924 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv47n\" (UniqueName: \"kubernetes.io/projected/10e2b910-b064-4774-91b6-b9f215f60e33-kube-api-access-fv47n\") pod \"iptables-alerter-hzv6w\" (UID: \"10e2b910-b064-4774-91b6-b9f215f60e33\") " pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.519719 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.519695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqqh5\" (UniqueName: \"kubernetes.io/projected/95fc751e-3c56-4f22-97c9-e0664b53b97f-kube-api-access-kqqh5\") pod \"multus-7tb9z\" (UID: \"95fc751e-3c56-4f22-97c9-e0664b53b97f\") " pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.520112 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.520092 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f26x6\" (UniqueName: \"kubernetes.io/projected/0d667da0-cb0d-4ae2-bfa5-d2f51069834e-kube-api-access-f26x6\") pod \"aws-ebs-csi-driver-node-rg2hb\" (UID: \"0d667da0-cb0d-4ae2-bfa5-d2f51069834e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.598797 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.598761 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" Aug 03 23:29:48.607764 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.607741 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7r62k" Aug 03 23:29:48.617535 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.617512 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nsl2l" Aug 03 23:29:48.625069 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.625050 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" Aug 03 23:29:48.631729 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.631707 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:29:48.639237 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.639217 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" Aug 03 23:29:48.646044 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.646026 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7tb9z" Aug 03 23:29:48.652638 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.652619 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:29:48.660170 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.660152 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hzv6w" Aug 03 23:29:48.905841 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.905769 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:48.905997 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:48.905845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:48.905997 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.905931 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:48.905997 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.905944 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:48.905997 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.905997 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:29:49.905976506 +0000 UTC m=+3.969232934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:48.906217 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:48.906014 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret podName:36bfdb11-2080-4407-9208-2d5ef6995e91 nodeName:}" failed. No retries permitted until 2026-08-03 23:29:49.906007503 +0000 UTC m=+3.969263910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret") pod "global-pull-secret-syncer-nd4xg" (UID: "36bfdb11-2080-4407-9208-2d5ef6995e91") : object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:49.006971 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:49.006940 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:49.007134 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:49.007073 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 03 23:29:49.007134 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:49.007092 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 03 23:29:49.007134 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:49.007101 2556 projected.go:196] Error preparing data for projected volume kube-api-access-2c84m for pod openshift-network-diagnostics/network-check-target-t7st8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:49.007288 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:49.007147 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m podName:57270cca-6d1b-4419-a389-ecdd66c5c92a nodeName:}" failed. No retries permitted until 2026-08-03 23:29:50.007133907 +0000 UTC m=+4.070390316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c84m" (UniqueName: "kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m") pod "network-check-target-t7st8" (UID: "57270cca-6d1b-4419-a389-ecdd66c5c92a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:49.331247 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:49.331144 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-02 23:24:48 +0000 UTC" deadline="2028-04-20 18:12:38.12862561 +0000 UTC" Aug 03 23:29:49.331247 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:49.331201 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15018h42m48.797432248s" Aug 03 23:29:49.518655 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:49.518630 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e2b910_b064_4774_91b6_b9f215f60e33.slice/crio-0cb55fad2ad4aa2a088b884233e04a16a6c434455067ed613c49d8cf2f8e9110 WatchSource:0}: Error finding container 0cb55fad2ad4aa2a088b884233e04a16a6c434455067ed613c49d8cf2f8e9110: Status 404 returned error can't find the container with id 0cb55fad2ad4aa2a088b884233e04a16a6c434455067ed613c49d8cf2f8e9110 Aug 03 23:29:49.519520 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:49.519494 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fc751e_3c56_4f22_97c9_e0664b53b97f.slice/crio-e5dd708f06630b01107a08f97a295b8254c529c2224b2117516f850f09af8b2f WatchSource:0}: Error finding container e5dd708f06630b01107a08f97a295b8254c529c2224b2117516f850f09af8b2f: Status 404 returned error can't find the container with id e5dd708f06630b01107a08f97a295b8254c529c2224b2117516f850f09af8b2f Aug 03 23:29:49.520757 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:49.520371 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d667da0_cb0d_4ae2_bfa5_d2f51069834e.slice/crio-1bd4165cdca73fccb872e0df96bef384a70ac66465ada0374ff15119657e36c9 WatchSource:0}: Error finding container 1bd4165cdca73fccb872e0df96bef384a70ac66465ada0374ff15119657e36c9: Status 404 returned error can't find the container with id 1bd4165cdca73fccb872e0df96bef384a70ac66465ada0374ff15119657e36c9 Aug 03 23:29:49.525951 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:29:49.525871 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaa22d4_2499_45d2_9a81_1c2aaa7cce02.slice/crio-c366b4e95471a69abbdb84fb6f06002faeccf49a52b0370cc8824fc71fe55950 WatchSource:0}: Error finding container c366b4e95471a69abbdb84fb6f06002faeccf49a52b0370cc8824fc71fe55950: Status 404 returned error can't find the container with id c366b4e95471a69abbdb84fb6f06002faeccf49a52b0370cc8824fc71fe55950 Aug 03 23:29:49.913393 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:49.913101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:49.913393 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:49.913379 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:49.913632 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:49.913246 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:49.913632 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:49.913489 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:49.913632 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:49.913506 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret podName:36bfdb11-2080-4407-9208-2d5ef6995e91 nodeName:}" failed. No retries permitted until 2026-08-03 23:29:51.913480269 +0000 UTC m=+5.976736693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret") pod "global-pull-secret-syncer-nd4xg" (UID: "36bfdb11-2080-4407-9208-2d5ef6995e91") : object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:49.913632 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:49.913535 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:29:51.913523634 +0000 UTC m=+5.976780041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:50.014551 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.014462 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:50.014736 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:50.014628 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 03 23:29:50.014736 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:50.014649 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 03 23:29:50.014736 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:50.014661 2556 projected.go:196] Error preparing data for projected volume kube-api-access-2c84m for pod openshift-network-diagnostics/network-check-target-t7st8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:50.014736 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:50.014722 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m podName:57270cca-6d1b-4419-a389-ecdd66c5c92a nodeName:}" failed. No retries permitted until 2026-08-03 23:29:52.014704663 +0000 UTC m=+6.077961084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c84m" (UniqueName: "kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m") pod "network-check-target-t7st8" (UID: "57270cca-6d1b-4419-a389-ecdd66c5c92a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:50.332681 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.332552 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-02 23:24:48 +0000 UTC" deadline="2028-05-21 12:31:48.766299102 +0000 UTC" Aug 03 23:29:50.332681 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.332607 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15757h1m58.433696173s" Aug 03 23:29:50.416089 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.415425 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:50.416089 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:50.415560 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:29:50.416089 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.415950 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:50.416089 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:50.416040 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:29:50.416765 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.416608 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:50.416765 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:50.416713 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:29:50.434436 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.434409 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" event={"ID":"73bcb60f-5b25-4b51-8d80-4bc2f5276f12","Type":"ContainerStarted","Data":"d9336babd74d8470462ec81cd6d111af53546e6f8d4ed958ae5a9b10ccd8ef11"} Aug 03 23:29:50.437875 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.437848 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zqgb" event={"ID":"d8903faa-2d47-47a2-a7c0-ef8d5f7fb148","Type":"ContainerStarted","Data":"9cf376ceb94d523b307750e7d9197220df33d877153522bf8731607644718917"} Aug 03 23:29:50.446247 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.446219 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" event={"ID":"d8b7e0082a8d432502c091a2ad7a9c78","Type":"ContainerStarted","Data":"911454c539d72c95010154840884061402054da0ec8cc7e88504648519616c35"} Aug 03 23:29:50.457685 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.457660 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nsl2l" event={"ID":"30f44684-cb92-47de-9a0f-ffc6ca6346e6","Type":"ContainerStarted","Data":"26f14f9c4c60b7db9e102c75f7ad18e5d98167eba99d1445060b2393af603450"} Aug 03 23:29:50.460093 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.460066 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"f34f8ceb31b7df897811a83a3122923780a4bd9fb40d4d4d7fcb3c312954e7e3"} Aug 03 23:29:50.469279 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.469252 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7r62k" event={"ID":"347d49d7-131d-4942-82ea-6ce44f73495d","Type":"ContainerStarted","Data":"63d00dbfb034c32b9b90f4e134f1b8b80bc9588d5f97a2815adc35f993d20cf5"} Aug 03 23:29:50.472842 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.472820 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" event={"ID":"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02","Type":"ContainerStarted","Data":"c366b4e95471a69abbdb84fb6f06002faeccf49a52b0370cc8824fc71fe55950"} Aug 03 23:29:50.476489 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.476465 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" event={"ID":"0d667da0-cb0d-4ae2-bfa5-d2f51069834e","Type":"ContainerStarted","Data":"1bd4165cdca73fccb872e0df96bef384a70ac66465ada0374ff15119657e36c9"} Aug 03 23:29:50.481139 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.481114 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7tb9z" event={"ID":"95fc751e-3c56-4f22-97c9-e0664b53b97f","Type":"ContainerStarted","Data":"e5dd708f06630b01107a08f97a295b8254c529c2224b2117516f850f09af8b2f"} Aug 03 23:29:50.483769 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.483746 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hzv6w" event={"ID":"10e2b910-b064-4774-91b6-b9f215f60e33","Type":"ContainerStarted","Data":"0cb55fad2ad4aa2a088b884233e04a16a6c434455067ed613c49d8cf2f8e9110"} Aug 03 23:29:50.492572 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.492548 2556 generic.go:356] "Generic (PLEG): container finished" podID="eba9639115ca8e3bac669ba1dab6cd0e" containerID="6858873b1d2c32b715b253cf0c271d8ba5435e55fcbdc9d4d3e0ba8144118f40" exitCode=0 Aug 03 23:29:50.492662 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.492583 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"eba9639115ca8e3bac669ba1dab6cd0e","Type":"ContainerDied","Data":"6858873b1d2c32b715b253cf0c271d8ba5435e55fcbdc9d4d3e0ba8144118f40"} Aug 03 23:29:50.507965 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:50.507918 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" podStartSLOduration=3.507900521 podStartE2EDuration="3.507900521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:29:50.46149384 +0000 UTC m=+4.524750271" watchObservedRunningTime="2026-08-03 23:29:50.507900521 +0000 UTC m=+4.571156955" Aug 03 23:29:51.511505 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:51.511465 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"eba9639115ca8e3bac669ba1dab6cd0e","Type":"ContainerStarted","Data":"b744564876c6f0f042a8842d0667b46a53996973212f2133a831a573f8ce806f"} Aug 03 23:29:51.939976 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:51.939889 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:51.940119 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:51.939986 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:51.940119 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:51.940109 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:51.940224 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:51.940168 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret podName:36bfdb11-2080-4407-9208-2d5ef6995e91 nodeName:}" failed. No retries permitted until 2026-08-03 23:29:55.940148853 +0000 UTC m=+10.003405278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret") pod "global-pull-secret-syncer-nd4xg" (UID: "36bfdb11-2080-4407-9208-2d5ef6995e91") : object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:51.940599 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:51.940576 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:51.940716 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:51.940633 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:29:55.940618676 +0000 UTC m=+10.003875086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:52.040433 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:52.040399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:52.040615 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:52.040541 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 03 23:29:52.040615 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:52.040568 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 03 23:29:52.040615 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:52.040580 2556 projected.go:196] Error preparing data for projected volume kube-api-access-2c84m for pod openshift-network-diagnostics/network-check-target-t7st8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:52.040775 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:52.040631 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m podName:57270cca-6d1b-4419-a389-ecdd66c5c92a nodeName:}" failed. No retries permitted until 2026-08-03 23:29:56.040612864 +0000 UTC m=+10.103869272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c84m" (UniqueName: "kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m") pod "network-check-target-t7st8" (UID: "57270cca-6d1b-4419-a389-ecdd66c5c92a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:52.414743 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:52.414664 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:52.414894 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:52.414670 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:52.414894 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:52.414808 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:29:52.414894 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:52.414821 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:52.415063 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:52.414904 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:29:52.415063 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:52.415003 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:29:54.418453 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:54.417594 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:54.418453 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:54.417721 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:29:54.418453 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:54.418086 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:54.418453 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:54.418175 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:29:54.418453 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:54.418229 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:54.418453 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:54.418330 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:29:55.974258 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:55.974221 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:55.974709 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:55.974284 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:55.974709 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:55.974469 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:55.974709 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:55.974530 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:03.97451045 +0000 UTC m=+18.037766871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:29:55.974946 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:55.974923 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:55.975109 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:55.974985 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret podName:36bfdb11-2080-4407-9208-2d5ef6995e91 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:03.974969148 +0000 UTC m=+18.038225570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret") pod "global-pull-secret-syncer-nd4xg" (UID: "36bfdb11-2080-4407-9208-2d5ef6995e91") : object "kube-system"/"original-pull-secret" not registered Aug 03 23:29:56.074779 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:56.074742 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:56.074970 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:56.074894 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 03 23:29:56.074970 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:56.074916 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 03 23:29:56.074970 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:56.074925 2556 projected.go:196] Error preparing data for projected volume kube-api-access-2c84m for pod openshift-network-diagnostics/network-check-target-t7st8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:56.074970 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:56.074968 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m podName:57270cca-6d1b-4419-a389-ecdd66c5c92a nodeName:}" failed. No retries permitted until 2026-08-03 23:30:04.074955343 +0000 UTC m=+18.138211750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c84m" (UniqueName: "kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m") pod "network-check-target-t7st8" (UID: "57270cca-6d1b-4419-a389-ecdd66c5c92a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:29:56.415914 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:56.415828 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:56.416065 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:56.415927 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:29:56.416295 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:56.416277 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:56.416391 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:56.416381 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:29:56.416511 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:56.416496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:56.416582 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:56.416568 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:29:58.415067 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:58.415028 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:29:58.415558 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:58.415141 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:29:58.415634 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:58.415586 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:29:58.415690 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:58.415675 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:29:58.415794 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:29:58.415778 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:29:58.415899 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:29:58.415877 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:00.414910 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:00.414879 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:00.414910 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:00.414901 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:00.415375 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:00.414879 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:00.415375 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:00.414989 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:00.415375 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:00.415073 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:00.415375 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:00.415111 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:02.415426 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:02.414964 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:02.415426 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:02.414969 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:02.415426 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:02.415072 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:02.415426 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:02.415097 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:02.415426 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:02.415162 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:02.415426 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:02.415282 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:04.034615 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:04.034582 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:04.035205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:04.034657 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:04.035205 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.034752 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:30:04.035205 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.034826 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:20.034807816 +0000 UTC m=+34.098064233 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:30:04.035205 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.034759 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 03 23:30:04.035205 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.034930 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret podName:36bfdb11-2080-4407-9208-2d5ef6995e91 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:20.034912704 +0000 UTC m=+34.098169111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret") pod "global-pull-secret-syncer-nd4xg" (UID: "36bfdb11-2080-4407-9208-2d5ef6995e91") : object "kube-system"/"original-pull-secret" not registered Aug 03 23:30:04.135667 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:04.135631 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:04.135863 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.135789 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 03 23:30:04.135863 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.135811 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 03 23:30:04.135863 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.135821 2556 projected.go:196] Error preparing data for projected volume kube-api-access-2c84m for pod openshift-network-diagnostics/network-check-target-t7st8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:30:04.136021 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.135878 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m podName:57270cca-6d1b-4419-a389-ecdd66c5c92a nodeName:}" failed. No retries permitted until 2026-08-03 23:30:20.135859219 +0000 UTC m=+34.199115629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c84m" (UniqueName: "kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m") pod "network-check-target-t7st8" (UID: "57270cca-6d1b-4419-a389-ecdd66c5c92a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:30:04.415693 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:04.415612 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:04.415850 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.415722 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:04.415850 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:04.415766 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:04.415958 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.415875 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:04.415958 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:04.415894 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:04.416023 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:04.415997 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:06.416136 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:06.416097 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:06.416769 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:06.416168 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:06.416896 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:06.416780 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:06.416896 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:06.416845 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:06.416896 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:06.416201 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:06.417064 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:06.417000 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:08.415470 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:08.415090 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:08.415470 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:08.415123 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:08.415470 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:08.415090 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:08.415470 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:08.415216 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:08.415470 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:08.415353 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:08.415470 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:08.415409 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:09.551491 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.551034 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nsl2l" event={"ID":"30f44684-cb92-47de-9a0f-ffc6ca6346e6","Type":"ContainerStarted","Data":"86f984d0ff9e43ffcbde3927f0d2bef665dffea4f64d9e3bad3d942f90440f3a"} Aug 03 23:30:09.554169 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.554140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"64157ab0a202c997d737c583bf4c2e8620df3d796a12b288553e54f81a52ef9c"} Aug 03 23:30:09.554323 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.554226 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"739460f237df5ac9c26bef23e78fbb0ad2d2149ba35b3abab49c8dc2e6538701"} Aug 03 23:30:09.554323 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.554272 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"a03931d350750a3004c9ada89d8f91d1f73bcf60d28eeed45c03ef57363a081c"} Aug 03 23:30:09.554431 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.554332 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"7198ca35b66154c6266889da8e9643403bf84ea5382109d2b3201395593755c4"} Aug 03 23:30:09.554431 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.554378 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"602269db6489f8ac6316a20744478046ff1c839082ccd5b5195b81b942aa9b60"} Aug 03 23:30:09.554431 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.554424 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"28940f85244fe124c363e7e2cfe88db1a703aee8e135299708b606734dd47ae6"} Aug 03 23:30:09.555711 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.555675 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7r62k" event={"ID":"347d49d7-131d-4942-82ea-6ce44f73495d","Type":"ContainerStarted","Data":"6424b65e4c7b8f19e70f14df1b41a87057eec920b7079c48c11812dd2d7716ec"} Aug 03 23:30:09.557439 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.557401 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" event={"ID":"4aaa22d4-2499-45d2-9a81-1c2aaa7cce02","Type":"ContainerStarted","Data":"e9b1771817938a953b00cff84e99b01f2c4af1affec041baf6d37faf038431a5"} Aug 03 23:30:09.559192 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.559165 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" event={"ID":"0d667da0-cb0d-4ae2-bfa5-d2f51069834e","Type":"ContainerStarted","Data":"896d3ab87218a823a498a4217a32c31e229721ab2efb8787903d6ba7a239c7e1"} Aug 03 23:30:09.560760 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.560739 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7tb9z" event={"ID":"95fc751e-3c56-4f22-97c9-e0664b53b97f","Type":"ContainerStarted","Data":"7eb488cf46dc0b14aacbd5a9bb464fa968e354ec2f74d21d8fe2cccd44b0f7b7"} Aug 03 23:30:09.562446 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.562425 2556 generic.go:356] "Generic (PLEG): container finished" podID="73bcb60f-5b25-4b51-8d80-4bc2f5276f12" containerID="f579580c9df49a53dfdf095cdb3aec89ffd07eb06e4ac22277d87d0fa95fdd0b" exitCode=0 Aug 03 23:30:09.562541 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.562488 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" event={"ID":"73bcb60f-5b25-4b51-8d80-4bc2f5276f12","Type":"ContainerDied","Data":"f579580c9df49a53dfdf095cdb3aec89ffd07eb06e4ac22277d87d0fa95fdd0b"} Aug 03 23:30:09.564013 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.563977 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zqgb" event={"ID":"d8903faa-2d47-47a2-a7c0-ef8d5f7fb148","Type":"ContainerStarted","Data":"0baf86dba7c04976e288e71fd2fdf72b7fb38ec0bf5b1425763fd57caa2acc1e"} Aug 03 23:30:09.566883 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.566839 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nsl2l" podStartSLOduration=4.736008751 podStartE2EDuration="23.566823252s" totalImagesPullingTime="18.830814501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:09.566418091 +0000 UTC m=+23.629674524" watchObservedRunningTime="2026-08-03 23:30:09.566823252 +0000 UTC m=+23.630079701" Aug 03 23:30:09.566979 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.566916 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" podStartSLOduration=22.566912271 podStartE2EDuration="22.566912271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:29:51.527184231 +0000 UTC m=+5.590440663" watchObservedRunningTime="2026-08-03 23:30:09.566912271 +0000 UTC m=+23.630168701" Aug 03 23:30:09.582527 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.582493 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7tb9z" podStartSLOduration=4.659772405 podStartE2EDuration="23.582480194s" totalImagesPullingTime="18.922707789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:09.582170969 +0000 UTC m=+23.645427399" watchObservedRunningTime="2026-08-03 23:30:09.582480194 +0000 UTC m=+23.645736623" Aug 03 23:30:09.618214 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.618175 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7r62k" podStartSLOduration=4.762723817 podStartE2EDuration="23.618157796s" totalImagesPullingTime="18.855433979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:09.617556854 +0000 UTC m=+23.680813285" watchObservedRunningTime="2026-08-03 23:30:09.618157796 +0000 UTC m=+23.681414227" Aug 03 23:30:09.634446 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.634411 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6zhpg" podStartSLOduration=4.753093444 podStartE2EDuration="23.634398239s" totalImagesPullingTime="18.881304795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:09.633816357 +0000 UTC m=+23.697072788" watchObservedRunningTime="2026-08-03 23:30:09.634398239 +0000 UTC m=+23.697654672" Aug 03 23:30:09.647468 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:09.647432 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4zqgb" podStartSLOduration=4.810675221 podStartE2EDuration="23.64741423s" totalImagesPullingTime="18.836739009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:09.646960569 +0000 UTC m=+23.710217002" watchObservedRunningTime="2026-08-03 23:30:09.64741423 +0000 UTC m=+23.710670662" Aug 03 23:30:10.133574 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.133548 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 03 23:30:10.349126 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.348979 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-03T23:30:10.133571986Z","UUID":"0fc8312f-4c52-4069-a90e-d4e4ee4f38d3","Handler":null,"Name":"","Endpoint":""} Aug 03 23:30:10.350906 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.350885 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 03 23:30:10.351054 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.350929 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 03 23:30:10.414722 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.414694 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:10.414879 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.414702 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:10.414879 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:10.414786 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:10.414879 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:10.414869 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:10.414980 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.414914 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:10.415012 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:10.414977 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:10.567953 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.567911 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" event={"ID":"0d667da0-cb0d-4ae2-bfa5-d2f51069834e","Type":"ContainerStarted","Data":"f71574bef912af884662a4a8b8f20667f3d1e7e4a525cc66b4408ef8cead0855"} Aug 03 23:30:10.569906 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:10.569860 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hzv6w" event={"ID":"10e2b910-b064-4774-91b6-b9f215f60e33","Type":"ContainerStarted","Data":"050c72d49417172ac535312d83e4d5dc6b84a8e53b0a71fc924580bc1666e8f4"} Aug 03 23:30:11.575845 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:11.575754 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"c3be2980eacc69fc3734045ab52a0b19f53645aaf4d90657870c61caa00831a8"} Aug 03 23:30:12.415291 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:12.415259 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:12.415291 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:12.415288 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:12.415557 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:12.415258 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:12.415557 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:12.415403 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:12.415557 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:12.415467 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:12.415557 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:12.415512 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:12.580687 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:12.580645 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" event={"ID":"0d667da0-cb0d-4ae2-bfa5-d2f51069834e","Type":"ContainerStarted","Data":"ed54c0cf3a9cbcfaedf1c993964b707d81f7d35a1b106a1da21d377377495a98"} Aug 03 23:30:12.598626 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:12.598574 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rg2hb" podStartSLOduration=4.838012159 podStartE2EDuration="26.598560566s" totalImagesPullingTime="21.760548407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:12.59818658 +0000 UTC m=+26.661443011" watchObservedRunningTime="2026-08-03 23:30:12.598560566 +0000 UTC m=+26.661816996" Aug 03 23:30:12.598893 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:12.598863 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hzv6w" podStartSLOduration=7.750287842 podStartE2EDuration="26.598853161s" totalImagesPullingTime="18.848565319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:10.584245686 +0000 UTC m=+24.647502117" watchObservedRunningTime="2026-08-03 23:30:12.598853161 +0000 UTC m=+26.662109594" Aug 03 23:30:12.881453 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:12.881360 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:30:12.882017 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:12.881989 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:30:13.917829 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:13.917612 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:30:13.918238 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:13.918225 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4zqgb" Aug 03 23:30:14.414617 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.414593 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:14.414617 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.414608 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:14.414764 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.414608 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:14.414764 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:14.414699 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:14.414836 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:14.414773 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:14.414836 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:14.414816 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:14.591527 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.591451 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" event={"ID":"f168a0b1-53b0-42d8-9bd6-f2ac0fa5550e","Type":"ContainerStarted","Data":"47a7cd23f27dc4344d6cb3468d231143b3990a6f7221fcf059ca583f4257eaac"} Aug 03 23:30:14.591641 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.591604 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:30:14.591695 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.591674 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:30:14.591806 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.591791 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:30:14.610943 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.610913 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:30:14.611563 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.611538 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:30:14.618344 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:14.618286 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" podStartSLOduration=9.693523941 podStartE2EDuration="28.618272136s" totalImagesPullingTime="18.924748195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:14.616366127 +0000 UTC m=+28.679622557" watchObservedRunningTime="2026-08-03 23:30:14.618272136 +0000 UTC m=+28.681528567" Aug 03 23:30:15.595944 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:15.595735 2556 generic.go:356] "Generic (PLEG): container finished" podID="73bcb60f-5b25-4b51-8d80-4bc2f5276f12" containerID="6b43e4bc33e95a9191133b88fb1ab15253981ddeb611c1683820794e4694a336" exitCode=0 Aug 03 23:30:15.596850 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:15.595820 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" event={"ID":"73bcb60f-5b25-4b51-8d80-4bc2f5276f12","Type":"ContainerDied","Data":"6b43e4bc33e95a9191133b88fb1ab15253981ddeb611c1683820794e4694a336"} Aug 03 23:30:16.415700 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.415671 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:16.415700 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.415704 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:16.415959 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:16.415764 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:16.415959 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.415829 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:16.415959 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:16.415835 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:16.415959 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:16.415899 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:16.643801 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.643765 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nd4xg"] Aug 03 23:30:16.644166 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.643874 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:16.644166 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:16.643995 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:16.646242 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.646213 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7st8"] Aug 03 23:30:16.646841 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.646823 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:16.646938 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.646887 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jsxr8"] Aug 03 23:30:16.646989 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:16.646931 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:16.646989 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:16.646976 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:16.647158 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:16.647080 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:17.602225 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:17.602143 2556 generic.go:356] "Generic (PLEG): container finished" podID="73bcb60f-5b25-4b51-8d80-4bc2f5276f12" containerID="0ef58aa24d6e3e2b0402508f15e591ca1a911dc4bf5a0c18a222b7c5fa89eee0" exitCode=0 Aug 03 23:30:17.602225 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:17.602198 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" event={"ID":"73bcb60f-5b25-4b51-8d80-4bc2f5276f12","Type":"ContainerDied","Data":"0ef58aa24d6e3e2b0402508f15e591ca1a911dc4bf5a0c18a222b7c5fa89eee0"} Aug 03 23:30:18.415146 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:18.415118 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:18.415531 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:18.415164 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:18.415531 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:18.415118 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:18.415531 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:18.415241 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:18.415531 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:18.415323 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:18.415531 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:18.415403 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:18.607097 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:18.607018 2556 generic.go:356] "Generic (PLEG): container finished" podID="73bcb60f-5b25-4b51-8d80-4bc2f5276f12" containerID="275018f0fad3243cf2622184f68bdacf9a3edec751371bb0cc89700a4ad962b9" exitCode=0 Aug 03 23:30:18.607097 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:18.607055 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" event={"ID":"73bcb60f-5b25-4b51-8d80-4bc2f5276f12","Type":"ContainerDied","Data":"275018f0fad3243cf2622184f68bdacf9a3edec751371bb0cc89700a4ad962b9"} Aug 03 23:30:20.049161 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:20.048898 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:20.049593 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.049295 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 03 23:30:20.049593 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.049456 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret podName:36bfdb11-2080-4407-9208-2d5ef6995e91 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:52.049373197 +0000 UTC m=+66.112629607 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret") pod "global-pull-secret-syncer-nd4xg" (UID: "36bfdb11-2080-4407-9208-2d5ef6995e91") : object "kube-system"/"original-pull-secret" not registered Aug 03 23:30:20.050015 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:20.049987 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:20.050152 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.050139 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:30:20.050202 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.050195 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:52.050178473 +0000 UTC m=+66.113434895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 03 23:30:20.150387 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:20.150357 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:20.150568 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.150503 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 03 23:30:20.150568 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.150524 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 03 23:30:20.150568 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.150537 2556 projected.go:196] Error preparing data for projected volume kube-api-access-2c84m for pod openshift-network-diagnostics/network-check-target-t7st8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:30:20.150745 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.150593 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m podName:57270cca-6d1b-4419-a389-ecdd66c5c92a nodeName:}" failed. No retries permitted until 2026-08-03 23:30:52.150575415 +0000 UTC m=+66.213831837 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c84m" (UniqueName: "kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m") pod "network-check-target-t7st8" (UID: "57270cca-6d1b-4419-a389-ecdd66c5c92a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 03 23:30:20.415744 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:20.415604 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:20.415744 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:20.415626 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:20.415744 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:20.415633 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:20.415744 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.415723 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7st8" podUID="57270cca-6d1b-4419-a389-ecdd66c5c92a" Aug 03 23:30:20.416178 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.415807 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nd4xg" podUID="36bfdb11-2080-4407-9208-2d5ef6995e91" Aug 03 23:30:20.416178 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:20.415897 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:30:21.268183 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.268153 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeReady" Aug 03 23:30:21.268664 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.268301 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 03 23:30:21.302445 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.302413 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5"] Aug 03 23:30:21.305697 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.305677 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-575cb9dd4c-znjgv"] Aug 03 23:30:21.306544 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.305905 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5" Aug 03 23:30:21.307387 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.307350 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.309033 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.309009 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 03 23:30:21.309347 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.309325 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 03 23:30:21.309467 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.309451 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ld6jb\"" Aug 03 23:30:21.309837 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.309674 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 03 23:30:21.309837 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.309792 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m6v2t\"" Aug 03 23:30:21.310156 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.310129 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 03 23:30:21.310230 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.310202 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 03 23:30:21.315849 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.315829 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5"] Aug 03 23:30:21.316267 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.316095 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 03 23:30:21.318698 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.318676 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9wh8d"] Aug 03 23:30:21.320792 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.320772 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9wh8d" Aug 03 23:30:21.321332 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.321246 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w4tdk"] Aug 03 23:30:21.323043 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.323021 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 03 23:30:21.323156 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.323064 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.323156 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.323105 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 03 23:30:21.323421 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.323328 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4skmh\"" Aug 03 23:30:21.323528 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.323428 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 03 23:30:21.326367 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.326347 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 03 23:30:21.326722 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.326534 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 03 23:30:21.326722 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.326591 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rmmw5\"" Aug 03 23:30:21.331432 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.331415 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9wh8d"] Aug 03 23:30:21.333569 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.333545 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575cb9dd4c-znjgv"] Aug 03 23:30:21.336098 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.336077 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w4tdk"] Aug 03 23:30:21.459027 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.458987 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzk98\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-kube-api-access-rzk98\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.459027 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459030 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5610d18a-21f3-4cab-be04-b7a8119b6105-cert\") pod \"ingress-canary-9wh8d\" (UID: \"5610d18a-21f3-4cab-be04-b7a8119b6105\") " pod="openshift-ingress-canary/ingress-canary-9wh8d" Aug 03 23:30:21.459263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459051 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-certificates\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.459263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459102 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jw7\" (UniqueName: \"kubernetes.io/projected/c363730f-1c9a-43cc-8835-35962547d651-kube-api-access-m4jw7\") pod \"network-check-source-65bdbbff45-c6qx5\" (UID: \"c363730f-1c9a-43cc-8835-35962547d651\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5" Aug 03 23:30:21.459263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459123 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdkxq\" (UniqueName: \"kubernetes.io/projected/2062046e-c1ef-42fb-b341-fab17c10c653-kube-api-access-wdkxq\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.459263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-installation-pull-secrets\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.459263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wz4\" (UniqueName: \"kubernetes.io/projected/5610d18a-21f3-4cab-be04-b7a8119b6105-kube-api-access-g5wz4\") pod \"ingress-canary-9wh8d\" (UID: \"5610d18a-21f3-4cab-be04-b7a8119b6105\") " pod="openshift-ingress-canary/ingress-canary-9wh8d" Aug 03 23:30:21.459263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459227 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-image-registry-private-configuration\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.459554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459281 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a44e778-2db8-449e-889e-07b17c8f8abe-ca-trust-extracted\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.459554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459320 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2062046e-c1ef-42fb-b341-fab17c10c653-config-volume\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.459554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459343 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.459554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459369 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-trusted-ca\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.459554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459421 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.459554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459449 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-bound-sa-token\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.459554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.459479 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2062046e-c1ef-42fb-b341-fab17c10c653-tmp-dir\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.471698 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.471668 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zqgb_d8903faa-2d47-47a2-a7c0-ef8d5f7fb148/konnectivity-agent/0.log" limit=1024 Aug 03 23:30:21.560370 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560268 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jw7\" (UniqueName: \"kubernetes.io/projected/c363730f-1c9a-43cc-8835-35962547d651-kube-api-access-m4jw7\") pod \"network-check-source-65bdbbff45-c6qx5\" (UID: \"c363730f-1c9a-43cc-8835-35962547d651\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5" Aug 03 23:30:21.560370 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560328 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdkxq\" (UniqueName: \"kubernetes.io/projected/2062046e-c1ef-42fb-b341-fab17c10c653-kube-api-access-wdkxq\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.560370 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560361 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-installation-pull-secrets\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.560594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560383 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wz4\" (UniqueName: \"kubernetes.io/projected/5610d18a-21f3-4cab-be04-b7a8119b6105-kube-api-access-g5wz4\") pod \"ingress-canary-9wh8d\" (UID: \"5610d18a-21f3-4cab-be04-b7a8119b6105\") " pod="openshift-ingress-canary/ingress-canary-9wh8d" Aug 03 23:30:21.560594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560412 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-image-registry-private-configuration\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.560594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a44e778-2db8-449e-889e-07b17c8f8abe-ca-trust-extracted\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.560594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2062046e-c1ef-42fb-b341-fab17c10c653-config-volume\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.560594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.560594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-trusted-ca\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.560594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560541 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.560594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-bound-sa-token\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.560856 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560604 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2062046e-c1ef-42fb-b341-fab17c10c653-tmp-dir\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.560856 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560651 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk98\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-kube-api-access-rzk98\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.560856 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560673 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5610d18a-21f3-4cab-be04-b7a8119b6105-cert\") pod \"ingress-canary-9wh8d\" (UID: \"5610d18a-21f3-4cab-be04-b7a8119b6105\") " pod="openshift-ingress-canary/ingress-canary-9wh8d" Aug 03 23:30:21.560856 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560699 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-certificates\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.560991 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.560962 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a44e778-2db8-449e-889e-07b17c8f8abe-ca-trust-extracted\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.561074 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:21.561060 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:30:21.561141 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:21.561131 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:22.061111616 +0000 UTC m=+36.124368027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:30:21.561348 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.561325 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-certificates\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.561466 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:21.561430 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:30:21.561466 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:21.561453 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:30:21.561592 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:21.561517 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:30:22.061499068 +0000 UTC m=+36.124755493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:30:21.561864 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.561842 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2062046e-c1ef-42fb-b341-fab17c10c653-tmp-dir\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.562033 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.562010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2062046e-c1ef-42fb-b341-fab17c10c653-config-volume\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.562190 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.562170 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-trusted-ca\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.562601 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.562576 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zqgb_d8903faa-2d47-47a2-a7c0-ef8d5f7fb148/konnectivity-agent/0.log" limit=1024 Aug 03 23:30:21.565616 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.565592 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-installation-pull-secrets\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.565720 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.565641 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-image-registry-private-configuration\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.565720 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.565711 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5610d18a-21f3-4cab-be04-b7a8119b6105-cert\") pod \"ingress-canary-9wh8d\" (UID: \"5610d18a-21f3-4cab-be04-b7a8119b6105\") " pod="openshift-ingress-canary/ingress-canary-9wh8d" Aug 03 23:30:21.569742 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.569711 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jw7\" (UniqueName: \"kubernetes.io/projected/c363730f-1c9a-43cc-8835-35962547d651-kube-api-access-m4jw7\") pod \"network-check-source-65bdbbff45-c6qx5\" (UID: \"c363730f-1c9a-43cc-8835-35962547d651\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5" Aug 03 23:30:21.572778 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.572373 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wz4\" (UniqueName: \"kubernetes.io/projected/5610d18a-21f3-4cab-be04-b7a8119b6105-kube-api-access-g5wz4\") pod \"ingress-canary-9wh8d\" (UID: \"5610d18a-21f3-4cab-be04-b7a8119b6105\") " pod="openshift-ingress-canary/ingress-canary-9wh8d" Aug 03 23:30:21.575564 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.575516 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdkxq\" (UniqueName: \"kubernetes.io/projected/2062046e-c1ef-42fb-b341-fab17c10c653-kube-api-access-wdkxq\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:21.576891 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.576853 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzk98\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-kube-api-access-rzk98\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.576970 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.576954 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-bound-sa-token\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:21.623741 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.623708 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5" Aug 03 23:30:21.640554 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.640520 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9wh8d" Aug 03 23:30:21.833106 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:21.833014 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zqgb_d8903faa-2d47-47a2-a7c0-ef8d5f7fb148/konnectivity-agent/0.log" limit=1024 Aug 03 23:30:22.065517 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.065480 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:22.065517 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.065538 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:22.065759 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:22.065638 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:30:22.065759 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:22.065661 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:30:22.065759 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:22.065681 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:30:22.065759 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:22.065685 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:23.065672756 +0000 UTC m=+37.128929166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:30:22.065759 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:22.065733 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:30:23.065715857 +0000 UTC m=+37.128972266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:30:22.415687 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.415648 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:22.416426 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.415649 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:22.416426 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.415648 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:22.418192 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.418166 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 03 23:30:22.418346 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.418261 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 03 23:30:22.419343 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.419300 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n4wm4\"" Aug 03 23:30:22.419447 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:22.419370 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6szp\"" Aug 03 23:30:23.075083 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:23.074968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:23.075083 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:23.075023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:23.075274 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:23.075125 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:30:23.075274 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:23.075129 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:30:23.075274 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:23.075146 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:30:23.075274 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:23.075190 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:25.07516986 +0000 UTC m=+39.138426323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:30:23.075274 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:23.075207 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:30:25.075200311 +0000 UTC m=+39.138456720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:30:23.179464 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:23.179260 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5"] Aug 03 23:30:23.182188 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:23.182146 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9wh8d"] Aug 03 23:30:23.183513 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:30:23.183264 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc363730f_1c9a_43cc_8835_35962547d651.slice/crio-7c92f23bd59e9f3eb2cc178e97bb412e0623c9cf6c054699e221a0e4a24053f2 WatchSource:0}: Error finding container 7c92f23bd59e9f3eb2cc178e97bb412e0623c9cf6c054699e221a0e4a24053f2: Status 404 returned error can't find the container with id 7c92f23bd59e9f3eb2cc178e97bb412e0623c9cf6c054699e221a0e4a24053f2 Aug 03 23:30:23.186393 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:30:23.186368 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5610d18a_21f3_4cab_be04_b7a8119b6105.slice/crio-a734ca365d1d91216a237b82a80b31121b3794741577a84d22a17dd7511582c1 WatchSource:0}: Error finding container a734ca365d1d91216a237b82a80b31121b3794741577a84d22a17dd7511582c1: Status 404 returned error can't find the container with id a734ca365d1d91216a237b82a80b31121b3794741577a84d22a17dd7511582c1 Aug 03 23:30:23.621293 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:23.621203 2556 generic.go:356] "Generic (PLEG): container finished" podID="73bcb60f-5b25-4b51-8d80-4bc2f5276f12" containerID="57223e96a4ce68b02364b43485914d847509d5c835120a540313a27714afb2f1" exitCode=0 Aug 03 23:30:23.621293 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:23.621274 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" event={"ID":"73bcb60f-5b25-4b51-8d80-4bc2f5276f12","Type":"ContainerDied","Data":"57223e96a4ce68b02364b43485914d847509d5c835120a540313a27714afb2f1"} Aug 03 23:30:23.622425 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:23.622399 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9wh8d" event={"ID":"5610d18a-21f3-4cab-be04-b7a8119b6105","Type":"ContainerStarted","Data":"a734ca365d1d91216a237b82a80b31121b3794741577a84d22a17dd7511582c1"} Aug 03 23:30:23.623515 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:23.623491 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5" event={"ID":"c363730f-1c9a-43cc-8835-35962547d651","Type":"ContainerStarted","Data":"7c92f23bd59e9f3eb2cc178e97bb412e0623c9cf6c054699e221a0e4a24053f2"} Aug 03 23:30:24.630772 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.630730 2556 generic.go:356] "Generic (PLEG): container finished" podID="73bcb60f-5b25-4b51-8d80-4bc2f5276f12" containerID="cff0df002cd267cb6a63fee91bd803e8d561ebdf6025785f08ee2161533b7e94" exitCode=0 Aug 03 23:30:24.631228 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.630789 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" event={"ID":"73bcb60f-5b25-4b51-8d80-4bc2f5276f12","Type":"ContainerDied","Data":"cff0df002cd267cb6a63fee91bd803e8d561ebdf6025785f08ee2161533b7e94"} Aug 03 23:30:24.760475 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.760444 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw"] Aug 03 23:30:24.768533 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.768504 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:24.771430 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.771399 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw"] Aug 03 23:30:24.771773 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.771756 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-svkcp\"" Aug 03 23:30:24.772975 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.772035 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 03 23:30:24.772975 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.772252 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 03 23:30:24.893889 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.893719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7aec628-af56-43fe-968c-81a79d5517f1-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:24.893889 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.893785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:24.994240 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.994198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7aec628-af56-43fe-968c-81a79d5517f1-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:24.994463 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.994258 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:24.994463 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:24.994385 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:30:24.994463 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:24.994458 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:25.494438346 +0000 UTC m=+39.557694779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:30:24.995111 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:24.995079 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7aec628-af56-43fe-968c-81a79d5517f1-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:25.095684 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:25.095645 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:25.095893 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:25.095714 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:25.095893 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:25.095823 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:30:25.095893 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:25.095846 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:30:25.095893 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:25.095877 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:30:25.096041 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:25.095910 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:30:29.095888248 +0000 UTC m=+43.159144677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:30:25.096041 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:25.095929 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:29.095920879 +0000 UTC m=+43.159177288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:30:25.500211 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:25.500157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:25.500396 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:25.500353 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:30:25.500462 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:25.500443 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:26.500420045 +0000 UTC m=+40.563676456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:30:25.638332 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:25.638278 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" event={"ID":"73bcb60f-5b25-4b51-8d80-4bc2f5276f12","Type":"ContainerStarted","Data":"aef82228568d4347f90666ab19d81c31aa7df34f51a5127237c34269cc9bf5bc"} Aug 03 23:30:25.661009 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:25.660923 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dxfpf" podStartSLOduration=9.354825596 podStartE2EDuration="39.660905112s" totalImagesPullingTime="30.306079516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:25.659728595 +0000 UTC m=+39.722985026" watchObservedRunningTime="2026-08-03 23:30:25.660905112 +0000 UTC m=+39.724161542" Aug 03 23:30:26.508363 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:26.508297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:26.508576 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:26.508501 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:30:26.508648 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:26.508580 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:28.508562736 +0000 UTC m=+42.571819167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:30:28.526837 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:28.526595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:28.527244 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:28.526756 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:30:28.527244 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:28.526998 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:32.526976819 +0000 UTC m=+46.590233227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:30:28.646131 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:28.646087 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9wh8d" event={"ID":"5610d18a-21f3-4cab-be04-b7a8119b6105","Type":"ContainerStarted","Data":"2e235098b0bd41800a972058b0f174ef5c5636ad95fb46474765175f4b6f1afa"} Aug 03 23:30:28.647695 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:28.647669 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5" event={"ID":"c363730f-1c9a-43cc-8835-35962547d651","Type":"ContainerStarted","Data":"e6077efc96cde081d843c938f12732c0ac99edaff37148e1afb12e1cbd94eb32"} Aug 03 23:30:28.663530 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:28.663433 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9wh8d" podStartSLOduration=3.146342791 podStartE2EDuration="7.66341798s" totalImagesPullingTime="4.517075189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:28.66189399 +0000 UTC m=+42.725150431" watchObservedRunningTime="2026-08-03 23:30:28.66341798 +0000 UTC m=+42.726674388" Aug 03 23:30:28.679824 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:28.679763 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-c6qx5" podStartSLOduration=26.165782082 podStartE2EDuration="30.679742111s" totalImagesPullingTime="4.513960029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:28.679029201 +0000 UTC m=+42.742285631" watchObservedRunningTime="2026-08-03 23:30:28.679742111 +0000 UTC m=+42.742998543" Aug 03 23:30:29.132386 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:29.132340 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:29.132567 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:29.132400 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:29.132567 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:29.132499 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:30:29.132567 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:29.132499 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:30:29.132567 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:29.132523 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:30:29.132567 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:29.132552 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:37.132539415 +0000 UTC m=+51.195795823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:30:29.132567 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:29.132565 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:30:37.13255966 +0000 UTC m=+51.195816068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:30:32.556124 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:32.556075 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:32.556631 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:32.556239 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:30:32.556631 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:32.556324 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:40.556290425 +0000 UTC m=+54.619546832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:30:37.188585 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:37.188538 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:37.188585 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:37.188597 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:37.189044 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:37.188704 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:30:37.189044 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:37.188702 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:30:37.189044 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:37.188727 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:30:37.189044 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:37.188757 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:53.188743301 +0000 UTC m=+67.251999709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:30:37.189044 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:37.188777 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:30:53.188763439 +0000 UTC m=+67.252019847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:30:40.617188 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:40.617123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:40.617688 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:40.617284 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:30:40.617688 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:40.617367 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:30:56.617350811 +0000 UTC m=+70.680607219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:30:46.621386 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:46.621354 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g9mb6" Aug 03 23:30:52.105620 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.105576 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:30:52.106075 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.105662 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:52.108150 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.108131 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 03 23:30:52.108241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.108225 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 03 23:30:52.116715 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:52.116674 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 03 23:30:52.116864 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:52.116766 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:31:56.116747207 +0000 UTC m=+130.180003615 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : secret "metrics-daemon-secret" not found Aug 03 23:30:52.120668 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.120642 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/36bfdb11-2080-4407-9208-2d5ef6995e91-original-pull-secret\") pod \"global-pull-secret-syncer-nd4xg\" (UID: \"36bfdb11-2080-4407-9208-2d5ef6995e91\") " pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:52.127692 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.127658 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nd4xg" Aug 03 23:30:52.206505 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.206467 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:52.209874 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.209850 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c84m\" (UniqueName: \"kubernetes.io/projected/57270cca-6d1b-4419-a389-ecdd66c5c92a-kube-api-access-2c84m\") pod \"network-check-target-t7st8\" (UID: \"57270cca-6d1b-4419-a389-ecdd66c5c92a\") " pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:52.270649 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.270613 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nd4xg"] Aug 03 23:30:52.273580 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:30:52.273549 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bfdb11_2080_4407_9208_2d5ef6995e91.slice/crio-4fc5270c0de706043a7b1bddb1c9caaa9a2cb25438a49762befe8d6d9b79f11a WatchSource:0}: Error finding container 4fc5270c0de706043a7b1bddb1c9caaa9a2cb25438a49762befe8d6d9b79f11a: Status 404 returned error can't find the container with id 4fc5270c0de706043a7b1bddb1c9caaa9a2cb25438a49762befe8d6d9b79f11a Aug 03 23:30:52.444205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.444109 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n4wm4\"" Aug 03 23:30:52.452207 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.452172 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:52.580026 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.579994 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7st8"] Aug 03 23:30:52.583266 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:30:52.583232 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57270cca_6d1b_4419_a389_ecdd66c5c92a.slice/crio-3a1fb73d339bc6679df7598f0b1f00add5df1edcfaccb6fff8b4a1a946f6df0e WatchSource:0}: Error finding container 3a1fb73d339bc6679df7598f0b1f00add5df1edcfaccb6fff8b4a1a946f6df0e: Status 404 returned error can't find the container with id 3a1fb73d339bc6679df7598f0b1f00add5df1edcfaccb6fff8b4a1a946f6df0e Aug 03 23:30:52.703897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.703799 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7st8" event={"ID":"57270cca-6d1b-4419-a389-ecdd66c5c92a","Type":"ContainerStarted","Data":"44184fb1077ea5fde87054cd4140fb29675752da999c235a8c0f1e88fc4d3810"} Aug 03 23:30:52.703897 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.703852 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7st8" event={"ID":"57270cca-6d1b-4419-a389-ecdd66c5c92a","Type":"ContainerStarted","Data":"3a1fb73d339bc6679df7598f0b1f00add5df1edcfaccb6fff8b4a1a946f6df0e"} Aug 03 23:30:52.704111 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.703909 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:30:52.705028 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.704994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nd4xg" event={"ID":"36bfdb11-2080-4407-9208-2d5ef6995e91","Type":"ContainerStarted","Data":"4fc5270c0de706043a7b1bddb1c9caaa9a2cb25438a49762befe8d6d9b79f11a"} Aug 03 23:30:52.719906 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:52.719851 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t7st8" podStartSLOduration=66.719831108 podStartE2EDuration="1m6.719831108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:52.719237876 +0000 UTC m=+66.782494309" watchObservedRunningTime="2026-08-03 23:30:52.719831108 +0000 UTC m=+66.783087537" Aug 03 23:30:53.215166 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:53.215124 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:30:53.215676 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:53.215221 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:30:53.215676 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:53.215328 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:30:53.215676 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:53.215357 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:30:53.215676 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:53.215394 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:30:53.215676 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:53.215437 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:31:25.215413666 +0000 UTC m=+99.278670081 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:30:53.215676 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:53.215457 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:31:25.215447925 +0000 UTC m=+99.278704334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:30:56.644197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:56.644097 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:30:56.644622 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:56.644262 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:30:56.644622 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:30:56.644356 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:31:28.644339499 +0000 UTC m=+102.707595907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:30:56.717049 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:56.717010 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nd4xg" event={"ID":"36bfdb11-2080-4407-9208-2d5ef6995e91","Type":"ContainerStarted","Data":"c657799de4acea32019cd58fa0a4c8c617c2cc1cb742cc61bba1bacdeaeb2b9f"} Aug 03 23:30:56.731791 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:30:56.731732 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nd4xg" podStartSLOduration=66.698521823 podStartE2EDuration="1m10.731712366s" totalImagesPullingTime="4.033190543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:30:56.731611145 +0000 UTC m=+70.794867588" watchObservedRunningTime="2026-08-03 23:30:56.731712366 +0000 UTC m=+70.794968797" Aug 03 23:31:23.710099 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:31:23.710067 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t7st8" Aug 03 23:31:25.257553 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:31:25.257513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:31:25.257935 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:31:25.257584 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:31:25.257935 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:25.257677 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:31:25.257935 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:25.257687 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:31:25.257935 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:25.257701 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:31:25.257935 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:25.257745 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:32:29.257729008 +0000 UTC m=+163.320985419 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:31:25.257935 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:25.257763 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:32:29.257756717 +0000 UTC m=+163.321013124 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:31:28.683238 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:31:28.683198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:31:28.683644 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:28.683367 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:31:28.683644 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:28.683438 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:32:32.683417909 +0000 UTC m=+166.746674316 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:31:56.175315 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:31:56.175264 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:31:56.175797 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:56.175428 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 03 23:31:56.175797 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:31:56.175511 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:33:58.175493741 +0000 UTC m=+252.238750149 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : secret "metrics-daemon-secret" not found Aug 03 23:32:24.334529 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:24.334491 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" podUID="8a44e778-2db8-449e-889e-07b17c8f8abe" Aug 03 23:32:24.347635 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:24.347605 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w4tdk" podUID="2062046e-c1ef-42fb-b341-fab17c10c653" Aug 03 23:32:24.928482 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:32:24.928450 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:32:24.928647 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:32:24.928494 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4tdk" Aug 03 23:32:25.434844 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:25.434805 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:32:27.781393 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:27.781349 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" podUID="d7aec628-af56-43fe-968c-81a79d5517f1" Aug 03 23:32:27.934486 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:32:27.934452 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:32:29.307433 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:32:29.307398 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:32:29.307848 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:32:29.307447 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:32:29.307848 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:29.307546 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 03 23:32:29.307848 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:29.307569 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575cb9dd4c-znjgv: secret "image-registry-tls" not found Aug 03 23:32:29.307848 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:29.307628 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls podName:8a44e778-2db8-449e-889e-07b17c8f8abe nodeName:}" failed. No retries permitted until 2026-08-03 23:34:31.307613707 +0000 UTC m=+285.370870114 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls") pod "image-registry-575cb9dd4c-znjgv" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe") : secret "image-registry-tls" not found Aug 03 23:32:29.307848 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:29.307549 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 03 23:32:29.307848 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:29.307684 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls podName:2062046e-c1ef-42fb-b341-fab17c10c653 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:31.307670663 +0000 UTC m=+285.370927074 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls") pod "dns-default-w4tdk" (UID: "2062046e-c1ef-42fb-b341-fab17c10c653") : secret "dns-default-metrics-tls" not found Aug 03 23:32:32.733164 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:32:32.733113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:32:32.733638 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:32.733266 2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 03 23:32:32.733638 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:32:32.733346 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert podName:d7aec628-af56-43fe-968c-81a79d5517f1 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:34.733329945 +0000 UTC m=+288.796586356 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-9cxjw" (UID: "d7aec628-af56-43fe-968c-81a79d5517f1") : secret "networking-console-plugin-cert" not found Aug 03 23:32:37.415219 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:32:37.415121 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:33:58.216407 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:33:58.216368 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:33:58.216865 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:33:58.216490 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 03 23:33:58.216865 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:33:58.216549 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs podName:29374835-d52e-403b-9834-071b2f9bda81 nodeName:}" failed. No retries permitted until 2026-08-03 23:36:00.216530601 +0000 UTC m=+374.279787010 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs") pod "network-metrics-daemon-jsxr8" (UID: "29374835-d52e-403b-9834-071b2f9bda81") : secret "metrics-daemon-secret" not found Aug 03 23:34:06.858386 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.858293 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5"] Aug 03 23:34:06.860066 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.860052 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:06.862615 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.862590 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 03 23:34:06.862748 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.862600 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 03 23:34:06.862748 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.862634 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 03 23:34:06.863613 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.863597 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 03 23:34:06.863712 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.863603 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8bg7b\"" Aug 03 23:34:06.869921 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.869412 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd"] Aug 03 23:34:06.871240 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.871220 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc"] Aug 03 23:34:06.871360 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.871349 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd" Aug 03 23:34:06.873291 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.873273 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-zp9bh"] Aug 03 23:34:06.873426 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.873382 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:06.873865 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.873842 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 03 23:34:06.873979 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.873910 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pngl7\"" Aug 03 23:34:06.874093 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.874067 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 03 23:34:06.875091 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.875070 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5"] Aug 03 23:34:06.875217 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.875137 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:06.875729 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.875710 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 03 23:34:06.876140 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.876120 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pf4vd\"" Aug 03 23:34:06.876270 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.876129 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 03 23:34:06.876423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.876408 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 03 23:34:06.877912 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.877896 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 03 23:34:06.878115 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.878100 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 03 23:34:06.878204 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.878169 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ktq9m\"" Aug 03 23:34:06.878266 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.878228 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 03 23:34:06.878340 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.878273 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 03 23:34:06.883596 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.883577 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd"] Aug 03 23:34:06.883785 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.883672 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 03 23:34:06.884871 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.884753 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc"] Aug 03 23:34:06.885960 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.885945 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-zp9bh"] Aug 03 23:34:06.973888 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.973862 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4100d470-402e-4f60-aecc-06e52fc7140b-config\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:06.973996 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.973893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5bw\" (UniqueName: \"kubernetes.io/projected/4100d470-402e-4f60-aecc-06e52fc7140b-kube-api-access-lj5bw\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:06.973996 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.973923 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9kp\" (UniqueName: \"kubernetes.io/projected/c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5-kube-api-access-9x9kp\") pod \"volume-data-source-validator-5b8776669-bgstd\" (UID: \"c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd" Aug 03 23:34:06.973996 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.973986 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4100d470-402e-4f60-aecc-06e52fc7140b-serving-cert\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:06.974091 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.974002 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:06.974091 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.974044 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb55347b-ee57-4828-a76c-2bc48d562e0e-config\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:06.974091 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.974069 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4100d470-402e-4f60-aecc-06e52fc7140b-trusted-ca\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:06.974179 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.974089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg28f\" (UniqueName: \"kubernetes.io/projected/994097ec-6192-478c-88fd-eeefc8a48818-kube-api-access-gg28f\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:06.974179 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.974122 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb55347b-ee57-4828-a76c-2bc48d562e0e-serving-cert\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:06.974179 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:06.974145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kzh\" (UniqueName: \"kubernetes.io/projected/cb55347b-ee57-4828-a76c-2bc48d562e0e-kube-api-access-96kzh\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:07.075220 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075179 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4100d470-402e-4f60-aecc-06e52fc7140b-config\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.075220 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5bw\" (UniqueName: \"kubernetes.io/projected/4100d470-402e-4f60-aecc-06e52fc7140b-kube-api-access-lj5bw\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.075410 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075254 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9kp\" (UniqueName: \"kubernetes.io/projected/c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5-kube-api-access-9x9kp\") pod \"volume-data-source-validator-5b8776669-bgstd\" (UID: \"c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd" Aug 03 23:34:07.075464 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4100d470-402e-4f60-aecc-06e52fc7140b-serving-cert\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.075501 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075483 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:07.075533 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075515 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb55347b-ee57-4828-a76c-2bc48d562e0e-config\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:07.075574 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075541 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4100d470-402e-4f60-aecc-06e52fc7140b-trusted-ca\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.075574 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg28f\" (UniqueName: \"kubernetes.io/projected/994097ec-6192-478c-88fd-eeefc8a48818-kube-api-access-gg28f\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:07.075674 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb55347b-ee57-4828-a76c-2bc48d562e0e-serving-cert\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:07.075674 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075652 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kzh\" (UniqueName: \"kubernetes.io/projected/cb55347b-ee57-4828-a76c-2bc48d562e0e-kube-api-access-96kzh\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:07.075674 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:07.075663 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 03 23:34:07.075817 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:07.075738 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls podName:994097ec-6192-478c-88fd-eeefc8a48818 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:07.575718506 +0000 UTC m=+261.638974913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-jszkc" (UID: "994097ec-6192-478c-88fd-eeefc8a48818") : secret "samples-operator-tls" not found Aug 03 23:34:07.075919 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.075887 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4100d470-402e-4f60-aecc-06e52fc7140b-config\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.076090 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.076071 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb55347b-ee57-4828-a76c-2bc48d562e0e-config\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:07.076694 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.076675 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4100d470-402e-4f60-aecc-06e52fc7140b-trusted-ca\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.077854 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.077831 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4100d470-402e-4f60-aecc-06e52fc7140b-serving-cert\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.078529 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.078511 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb55347b-ee57-4828-a76c-2bc48d562e0e-serving-cert\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:07.083964 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.083944 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kzh\" (UniqueName: \"kubernetes.io/projected/cb55347b-ee57-4828-a76c-2bc48d562e0e-kube-api-access-96kzh\") pod \"service-ca-operator-784f97bcf6-d98z5\" (UID: \"cb55347b-ee57-4828-a76c-2bc48d562e0e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:07.084474 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.084455 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5bw\" (UniqueName: \"kubernetes.io/projected/4100d470-402e-4f60-aecc-06e52fc7140b-kube-api-access-lj5bw\") pod \"console-operator-5ccd6b445f-zp9bh\" (UID: \"4100d470-402e-4f60-aecc-06e52fc7140b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.084537 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.084480 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9kp\" (UniqueName: \"kubernetes.io/projected/c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5-kube-api-access-9x9kp\") pod \"volume-data-source-validator-5b8776669-bgstd\" (UID: \"c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd" Aug 03 23:34:07.084641 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.084624 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg28f\" (UniqueName: \"kubernetes.io/projected/994097ec-6192-478c-88fd-eeefc8a48818-kube-api-access-gg28f\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:07.171091 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.171038 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" Aug 03 23:34:07.182600 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.182580 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd" Aug 03 23:34:07.196643 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.196623 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:07.309036 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.308881 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5"] Aug 03 23:34:07.313001 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:07.312972 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb55347b_ee57_4828_a76c_2bc48d562e0e.slice/crio-0df5e0e449ebd2312c1d3313ecb88c2c305547a2ce907c03e0273fcf7adc5d8d WatchSource:0}: Error finding container 0df5e0e449ebd2312c1d3313ecb88c2c305547a2ce907c03e0273fcf7adc5d8d: Status 404 returned error can't find the container with id 0df5e0e449ebd2312c1d3313ecb88c2c305547a2ce907c03e0273fcf7adc5d8d Aug 03 23:34:07.325365 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.325336 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd"] Aug 03 23:34:07.327929 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:07.327906 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39dd3f1_1988_4c1e_81ba_c57cfb0a75f5.slice/crio-1bdff1cb37280ebc0fbe31607293859e1d598888fc8bd2e3d9c4115307748c11 WatchSource:0}: Error finding container 1bdff1cb37280ebc0fbe31607293859e1d598888fc8bd2e3d9c4115307748c11: Status 404 returned error can't find the container with id 1bdff1cb37280ebc0fbe31607293859e1d598888fc8bd2e3d9c4115307748c11 Aug 03 23:34:07.341097 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.341078 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-zp9bh"] Aug 03 23:34:07.343202 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:07.343180 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4100d470_402e_4f60_aecc_06e52fc7140b.slice/crio-a43894d6e7e4b5de669505dd386cbab02d2a31eb83b8c41ae3d3d6cfdf83cb7d WatchSource:0}: Error finding container a43894d6e7e4b5de669505dd386cbab02d2a31eb83b8c41ae3d3d6cfdf83cb7d: Status 404 returned error can't find the container with id a43894d6e7e4b5de669505dd386cbab02d2a31eb83b8c41ae3d3d6cfdf83cb7d Aug 03 23:34:07.580009 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:07.579978 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:07.580167 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:07.580138 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 03 23:34:07.580215 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:07.580206 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls podName:994097ec-6192-478c-88fd-eeefc8a48818 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:08.580186915 +0000 UTC m=+262.643443326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-jszkc" (UID: "994097ec-6192-478c-88fd-eeefc8a48818") : secret "samples-operator-tls" not found Aug 03 23:34:08.171167 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:08.171086 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" event={"ID":"4100d470-402e-4f60-aecc-06e52fc7140b","Type":"ContainerStarted","Data":"a43894d6e7e4b5de669505dd386cbab02d2a31eb83b8c41ae3d3d6cfdf83cb7d"} Aug 03 23:34:08.172495 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:08.172459 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" event={"ID":"cb55347b-ee57-4828-a76c-2bc48d562e0e","Type":"ContainerStarted","Data":"0df5e0e449ebd2312c1d3313ecb88c2c305547a2ce907c03e0273fcf7adc5d8d"} Aug 03 23:34:08.174090 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:08.174062 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd" event={"ID":"c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5","Type":"ContainerStarted","Data":"1bdff1cb37280ebc0fbe31607293859e1d598888fc8bd2e3d9c4115307748c11"} Aug 03 23:34:08.589124 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:08.589088 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:08.589297 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:08.589196 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 03 23:34:08.589297 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:08.589264 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls podName:994097ec-6192-478c-88fd-eeefc8a48818 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:10.589245296 +0000 UTC m=+264.652501707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-jszkc" (UID: "994097ec-6192-478c-88fd-eeefc8a48818") : secret "samples-operator-tls" not found Aug 03 23:34:09.178318 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:09.178268 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd" event={"ID":"c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5","Type":"ContainerStarted","Data":"647ddc329fda9bec90c00d18482be4c6f2e8db8c2b4cea8690ad2d7a6a6832bc"} Aug 03 23:34:09.193847 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:09.193789 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-bgstd" podStartSLOduration=2.112691946 podStartE2EDuration="3.193771738s" totalImagesPullingTime="1.081079792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:09.193673463 +0000 UTC m=+263.256929896" watchObservedRunningTime="2026-08-03 23:34:09.193771738 +0000 UTC m=+263.257028147" Aug 03 23:34:10.183149 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.183056 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" event={"ID":"cb55347b-ee57-4828-a76c-2bc48d562e0e","Type":"ContainerStarted","Data":"534a93636f976f6c6df4e83d57215d624e783c13a2af22e0839ed573957db609"} Aug 03 23:34:10.184715 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.184691 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/0.log" Aug 03 23:34:10.184844 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.184732 2556 generic.go:356] "Generic (PLEG): container finished" podID="4100d470-402e-4f60-aecc-06e52fc7140b" containerID="e31b08714c0c05a9aaf9da098c6a93df71b2f1aef1b1c766b595b89962f54757" exitCode=255 Aug 03 23:34:10.184844 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.184803 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" event={"ID":"4100d470-402e-4f60-aecc-06e52fc7140b","Type":"ContainerDied","Data":"e31b08714c0c05a9aaf9da098c6a93df71b2f1aef1b1c766b595b89962f54757"} Aug 03 23:34:10.184993 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.184976 2556 scope.go:117] "RemoveContainer" containerID="e31b08714c0c05a9aaf9da098c6a93df71b2f1aef1b1c766b595b89962f54757" Aug 03 23:34:10.198890 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.198851 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" podStartSLOduration=1.6358985449999999 podStartE2EDuration="4.198837064s" totalImagesPullingTime="2.562938519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:10.198451437 +0000 UTC m=+264.261707866" watchObservedRunningTime="2026-08-03 23:34:10.198837064 +0000 UTC m=+264.262093492" Aug 03 23:34:10.604764 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.604727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:10.604919 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:10.604860 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 03 23:34:10.604919 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:10.604915 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls podName:994097ec-6192-478c-88fd-eeefc8a48818 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:14.604901707 +0000 UTC m=+268.668158114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-jszkc" (UID: "994097ec-6192-478c-88fd-eeefc8a48818") : secret "samples-operator-tls" not found Aug 03 23:34:10.839439 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.839402 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl"] Aug 03 23:34:10.842646 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.842627 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" Aug 03 23:34:10.845199 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.845175 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lld2r\"" Aug 03 23:34:10.846272 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.846254 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 03 23:34:10.846392 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.846270 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 03 23:34:10.851097 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.851075 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl"] Aug 03 23:34:10.908104 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:10.908040 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxg7\" (UniqueName: \"kubernetes.io/projected/5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe-kube-api-access-4bxg7\") pod \"migrator-68bc745d87-kghsl\" (UID: \"5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" Aug 03 23:34:11.009277 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.009249 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxg7\" (UniqueName: \"kubernetes.io/projected/5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe-kube-api-access-4bxg7\") pod \"migrator-68bc745d87-kghsl\" (UID: \"5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" Aug 03 23:34:11.017827 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.017796 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxg7\" (UniqueName: \"kubernetes.io/projected/5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe-kube-api-access-4bxg7\") pod \"migrator-68bc745d87-kghsl\" (UID: \"5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" Aug 03 23:34:11.152373 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.152344 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" Aug 03 23:34:11.190087 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.190064 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:34:11.190763 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.190747 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/0.log" Aug 03 23:34:11.190826 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.190783 2556 generic.go:356] "Generic (PLEG): container finished" podID="4100d470-402e-4f60-aecc-06e52fc7140b" containerID="bfd56ca41920bc6bb83e90a18eca5b473d885303ab4b2babf72120e84ebe28a6" exitCode=255 Aug 03 23:34:11.190888 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.190869 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" event={"ID":"4100d470-402e-4f60-aecc-06e52fc7140b","Type":"ContainerDied","Data":"bfd56ca41920bc6bb83e90a18eca5b473d885303ab4b2babf72120e84ebe28a6"} Aug 03 23:34:11.190929 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.190920 2556 scope.go:117] "RemoveContainer" containerID="e31b08714c0c05a9aaf9da098c6a93df71b2f1aef1b1c766b595b89962f54757" Aug 03 23:34:11.191151 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.191134 2556 scope.go:117] "RemoveContainer" containerID="bfd56ca41920bc6bb83e90a18eca5b473d885303ab4b2babf72120e84ebe28a6" Aug 03 23:34:11.191366 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:11.191342 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-zp9bh_openshift-console-operator(4100d470-402e-4f60-aecc-06e52fc7140b)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" podUID="4100d470-402e-4f60-aecc-06e52fc7140b" Aug 03 23:34:11.277678 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.277527 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl"] Aug 03 23:34:11.279873 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:11.279849 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4d21d3_12ef_4c7d_a4d0_72f5869b3dfe.slice/crio-56c9cff80143129bb0c581443932d9233c2a2f22f7545dfa720fddf5bbd0c6c8 WatchSource:0}: Error finding container 56c9cff80143129bb0c581443932d9233c2a2f22f7545dfa720fddf5bbd0c6c8: Status 404 returned error can't find the container with id 56c9cff80143129bb0c581443932d9233c2a2f22f7545dfa720fddf5bbd0c6c8 Aug 03 23:34:11.408955 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.408924 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5qhrx"] Aug 03 23:34:11.413452 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.413409 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.416148 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.416123 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 03 23:34:11.416243 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.416172 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 03 23:34:11.416243 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.416223 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 03 23:34:11.416243 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.416230 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 03 23:34:11.416410 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.416179 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lglgl\"" Aug 03 23:34:11.420343 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.420318 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5qhrx"] Aug 03 23:34:11.514271 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.514242 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95547717-0f6c-42dc-9cc2-219289a032f7-data-volume\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.514271 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.514273 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.514448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.514292 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s79nc\" (UniqueName: \"kubernetes.io/projected/95547717-0f6c-42dc-9cc2-219289a032f7-kube-api-access-s79nc\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.514448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.514367 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.514517 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.514503 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95547717-0f6c-42dc-9cc2-219289a032f7-crio-socket\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.615603 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.615576 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95547717-0f6c-42dc-9cc2-219289a032f7-data-volume\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.615721 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.615611 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.615721 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.615638 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s79nc\" (UniqueName: \"kubernetes.io/projected/95547717-0f6c-42dc-9cc2-219289a032f7-kube-api-access-s79nc\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.615721 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:11.615717 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 03 23:34:11.615860 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.615738 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.615860 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:11.615767 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:12.115753269 +0000 UTC m=+266.179009676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : secret "insights-runtime-extractor-tls" not found Aug 03 23:34:11.615860 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:11.615803 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 03 23:34:11.615860 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.615843 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95547717-0f6c-42dc-9cc2-219289a032f7-crio-socket\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.615860 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:11.615861 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:12.115842903 +0000 UTC m=+266.179099327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : configmap "kube-rbac-proxy" not found Aug 03 23:34:11.616042 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.615885 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95547717-0f6c-42dc-9cc2-219289a032f7-crio-socket\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.616042 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.615943 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95547717-0f6c-42dc-9cc2-219289a032f7-data-volume\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:11.624023 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:11.624003 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s79nc\" (UniqueName: \"kubernetes.io/projected/95547717-0f6c-42dc-9cc2-219289a032f7-kube-api-access-s79nc\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:12.120606 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:12.120564 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:12.120790 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:12.120619 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:12.120790 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:12.120707 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 03 23:34:12.120790 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:12.120741 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 03 23:34:12.120790 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:12.120772 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:13.120754439 +0000 UTC m=+267.184010858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : configmap "kube-rbac-proxy" not found Aug 03 23:34:12.120989 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:12.120800 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:13.120787316 +0000 UTC m=+267.184043723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : secret "insights-runtime-extractor-tls" not found Aug 03 23:34:12.195963 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:12.195931 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:34:12.196438 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:12.196297 2556 scope.go:117] "RemoveContainer" containerID="bfd56ca41920bc6bb83e90a18eca5b473d885303ab4b2babf72120e84ebe28a6" Aug 03 23:34:12.196510 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:12.196494 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-zp9bh_openshift-console-operator(4100d470-402e-4f60-aecc-06e52fc7140b)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" podUID="4100d470-402e-4f60-aecc-06e52fc7140b" Aug 03 23:34:12.197322 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:12.197284 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" event={"ID":"5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe","Type":"ContainerStarted","Data":"56c9cff80143129bb0c581443932d9233c2a2f22f7545dfa720fddf5bbd0c6c8"} Aug 03 23:34:13.127949 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.127915 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:13.127949 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.127958 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:13.128160 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:13.128031 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 03 23:34:13.128160 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:13.128047 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 03 23:34:13.128160 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:13.128087 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:15.12807028 +0000 UTC m=+269.191326689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : configmap "kube-rbac-proxy" not found Aug 03 23:34:13.128160 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:13.128102 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:15.128096399 +0000 UTC m=+269.191352807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : secret "insights-runtime-extractor-tls" not found Aug 03 23:34:13.202418 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.202387 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" event={"ID":"5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe","Type":"ContainerStarted","Data":"1d92e5cd089a0c9044826a84863ee2509889c1b114e05b600fa2b02a343f1e59"} Aug 03 23:34:13.202747 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.202444 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" event={"ID":"5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe","Type":"ContainerStarted","Data":"614e17b454fe252be7767723c8d74cc468d45af8b19c7b24142b8922c8a1be2e"} Aug 03 23:34:13.218930 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.218890 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-kghsl" podStartSLOduration=2.190184113 podStartE2EDuration="3.218876588s" totalImagesPullingTime="1.028692475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:13.21698455 +0000 UTC m=+267.280240993" watchObservedRunningTime="2026-08-03 23:34:13.218876588 +0000 UTC m=+267.282133017" Aug 03 23:34:13.285564 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.284328 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gm556"] Aug 03 23:34:13.288482 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.288459 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.290903 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.290882 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 03 23:34:13.291027 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.290976 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 03 23:34:13.291027 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.291005 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 03 23:34:13.291143 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.291005 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6wlgj\"" Aug 03 23:34:13.291143 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.291094 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 03 23:34:13.295609 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.295591 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gm556"] Aug 03 23:34:13.329556 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.329532 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cb2139b-cbe8-4340-b48b-52316c5c54af-signing-key\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.329649 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.329597 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cc5l\" (UniqueName: \"kubernetes.io/projected/7cb2139b-cbe8-4340-b48b-52316c5c54af-kube-api-access-7cc5l\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.329649 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.329625 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cb2139b-cbe8-4340-b48b-52316c5c54af-signing-cabundle\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.430153 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.430092 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cb2139b-cbe8-4340-b48b-52316c5c54af-signing-key\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.430263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.430155 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cc5l\" (UniqueName: \"kubernetes.io/projected/7cb2139b-cbe8-4340-b48b-52316c5c54af-kube-api-access-7cc5l\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.430263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.430181 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cb2139b-cbe8-4340-b48b-52316c5c54af-signing-cabundle\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.430829 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.430802 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cb2139b-cbe8-4340-b48b-52316c5c54af-signing-cabundle\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.432689 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.432667 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cb2139b-cbe8-4340-b48b-52316c5c54af-signing-key\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.438449 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.438429 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cc5l\" (UniqueName: \"kubernetes.io/projected/7cb2139b-cbe8-4340-b48b-52316c5c54af-kube-api-access-7cc5l\") pod \"service-ca-98c88fbb5-gm556\" (UID: \"7cb2139b-cbe8-4340-b48b-52316c5c54af\") " pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.598024 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.598000 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-gm556" Aug 03 23:34:13.717363 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.717243 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gm556"] Aug 03 23:34:13.719250 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:13.719230 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb2139b_cbe8_4340_b48b_52316c5c54af.slice/crio-9aa05e15bc928ebfda38dd14c40b69e26bf352742dacc3ef2a9e60f5abe1c677 WatchSource:0}: Error finding container 9aa05e15bc928ebfda38dd14c40b69e26bf352742dacc3ef2a9e60f5abe1c677: Status 404 returned error can't find the container with id 9aa05e15bc928ebfda38dd14c40b69e26bf352742dacc3ef2a9e60f5abe1c677 Aug 03 23:34:13.989664 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:13.989590 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7r62k_347d49d7-131d-4942-82ea-6ce44f73495d/dns-node-resolver/0.log" Aug 03 23:34:14.207273 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:14.207235 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-gm556" event={"ID":"7cb2139b-cbe8-4340-b48b-52316c5c54af","Type":"ContainerStarted","Data":"596ac5415e4c76d54848307c983981588828272aaf815111216df17582cdd2af"} Aug 03 23:34:14.207620 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:14.207295 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-gm556" event={"ID":"7cb2139b-cbe8-4340-b48b-52316c5c54af","Type":"ContainerStarted","Data":"9aa05e15bc928ebfda38dd14c40b69e26bf352742dacc3ef2a9e60f5abe1c677"} Aug 03 23:34:14.223672 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:14.223617 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-gm556" podStartSLOduration=1.223598805 podStartE2EDuration="1.223598805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:14.222931262 +0000 UTC m=+268.286187692" watchObservedRunningTime="2026-08-03 23:34:14.223598805 +0000 UTC m=+268.286855235" Aug 03 23:34:14.639378 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:14.639349 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:14.639570 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:14.639517 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 03 23:34:14.639638 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:14.639591 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls podName:994097ec-6192-478c-88fd-eeefc8a48818 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:22.639571672 +0000 UTC m=+276.702828081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-jszkc" (UID: "994097ec-6192-478c-88fd-eeefc8a48818") : secret "samples-operator-tls" not found Aug 03 23:34:14.989330 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:14.989276 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nsl2l_30f44684-cb92-47de-9a0f-ffc6ca6346e6/node-ca/0.log" Aug 03 23:34:15.142992 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:15.142956 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:15.143283 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:15.143125 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 03 23:34:15.143395 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:15.143369 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 03 23:34:15.143449 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:15.143426 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:19.143407268 +0000 UTC m=+273.206663687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : configmap "kube-rbac-proxy" not found Aug 03 23:34:15.143551 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:15.143272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:15.143834 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:15.143816 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:19.143791186 +0000 UTC m=+273.207047596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : secret "insights-runtime-extractor-tls" not found Aug 03 23:34:15.789026 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:15.788980 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9wh8d_5610d18a-21f3-4cab-be04-b7a8119b6105/serve-healthcheck-canary/0.log" Aug 03 23:34:16.391728 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:16.391681 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-kghsl_5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe/migrator/0.log" Aug 03 23:34:16.589225 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:16.589199 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-kghsl_5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe/graceful-termination/0.log" Aug 03 23:34:17.197886 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:17.197808 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:17.197886 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:17.197859 2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:17.198243 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:17.198182 2556 scope.go:117] "RemoveContainer" containerID="bfd56ca41920bc6bb83e90a18eca5b473d885303ab4b2babf72120e84ebe28a6" Aug 03 23:34:17.198361 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:17.198343 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-zp9bh_openshift-console-operator(4100d470-402e-4f60-aecc-06e52fc7140b)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" podUID="4100d470-402e-4f60-aecc-06e52fc7140b" Aug 03 23:34:19.176167 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:19.176127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:19.176551 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:19.176228 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:19.176551 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:19.176269 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 03 23:34:19.176551 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:19.176330 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 03 23:34:19.176551 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:19.176359 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:27.176344234 +0000 UTC m=+281.239600641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : configmap "kube-rbac-proxy" not found Aug 03 23:34:19.176551 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:19.176375 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls podName:95547717-0f6c-42dc-9cc2-219289a032f7 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:27.176368822 +0000 UTC m=+281.239625230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qhrx" (UID: "95547717-0f6c-42dc-9cc2-219289a032f7") : secret "insights-runtime-extractor-tls" not found Aug 03 23:34:22.703866 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:22.703832 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:22.706255 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:22.706234 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/994097ec-6192-478c-88fd-eeefc8a48818-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-jszkc\" (UID: \"994097ec-6192-478c-88fd-eeefc8a48818\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:22.790631 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:22.790602 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" Aug 03 23:34:22.908782 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:22.908746 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc"] Aug 03 23:34:23.236126 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:23.236100 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" event={"ID":"994097ec-6192-478c-88fd-eeefc8a48818","Type":"ContainerStarted","Data":"19267206a6e0cf1266cc8fbd83c59aa3b6663105e811ec4f68c1b45cd4bd83e1"} Aug 03 23:34:25.244865 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:25.244830 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" event={"ID":"994097ec-6192-478c-88fd-eeefc8a48818","Type":"ContainerStarted","Data":"5970dd9d4343b563e0d132b5c212628c704e6abeaf79915e1c8f787854d0ee7e"} Aug 03 23:34:25.245344 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:25.244910 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" event={"ID":"994097ec-6192-478c-88fd-eeefc8a48818","Type":"ContainerStarted","Data":"1edba2dc9549f0287178cce46d66595dd924a1737b78b04f7b7c4e3bf949ceab"} Aug 03 23:34:25.261444 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:25.261398 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-jszkc" podStartSLOduration=17.798000695 podStartE2EDuration="19.261384853s" totalImagesPullingTime="1.463384158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:25.260246709 +0000 UTC m=+279.323503137" watchObservedRunningTime="2026-08-03 23:34:25.261384853 +0000 UTC m=+279.324641304" Aug 03 23:34:27.238510 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:27.238478 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:27.238881 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:27.238518 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:27.238960 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:27.238942 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95547717-0f6c-42dc-9cc2-219289a032f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:27.240851 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:27.240831 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95547717-0f6c-42dc-9cc2-219289a032f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qhrx\" (UID: \"95547717-0f6c-42dc-9cc2-219289a032f7\") " pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:27.322983 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:27.322953 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qhrx" Aug 03 23:34:27.442962 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:27.442938 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5qhrx"] Aug 03 23:34:27.445572 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:27.445542 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95547717_0f6c_42dc_9cc2_219289a032f7.slice/crio-be13c0315a1e4f03e8186ab28299e6ebde2d7184753facaa213ce57cca8b93c6 WatchSource:0}: Error finding container be13c0315a1e4f03e8186ab28299e6ebde2d7184753facaa213ce57cca8b93c6: Status 404 returned error can't find the container with id be13c0315a1e4f03e8186ab28299e6ebde2d7184753facaa213ce57cca8b93c6 Aug 03 23:34:27.929646 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:27.929554 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" podUID="8a44e778-2db8-449e-889e-07b17c8f8abe" Aug 03 23:34:27.929646 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:27.929578 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-w4tdk" podUID="2062046e-c1ef-42fb-b341-fab17c10c653" Aug 03 23:34:28.254228 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:28.254200 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4tdk" Aug 03 23:34:28.254570 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:28.254226 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qhrx" event={"ID":"95547717-0f6c-42dc-9cc2-219289a032f7","Type":"ContainerStarted","Data":"c1262d5916dca35060ba1f6fd5c32d4ec6e62a37f27fde912cf3b224e8ac1a73"} Aug 03 23:34:28.254570 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:28.254300 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qhrx" event={"ID":"95547717-0f6c-42dc-9cc2-219289a032f7","Type":"ContainerStarted","Data":"c9e834b9065bafcbadb07295b9e3253a4e829aed4070fabbb4b3c87297dbaf38"} Aug 03 23:34:28.254570 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:28.254361 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qhrx" event={"ID":"95547717-0f6c-42dc-9cc2-219289a032f7","Type":"ContainerStarted","Data":"be13c0315a1e4f03e8186ab28299e6ebde2d7184753facaa213ce57cca8b93c6"} Aug 03 23:34:28.254570 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:28.254366 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:34:29.415768 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:29.415727 2556 scope.go:117] "RemoveContainer" containerID="bfd56ca41920bc6bb83e90a18eca5b473d885303ab4b2babf72120e84ebe28a6" Aug 03 23:34:30.261773 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:30.261746 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:34:30.261950 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:30.261846 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" event={"ID":"4100d470-402e-4f60-aecc-06e52fc7140b","Type":"ContainerStarted","Data":"93e853089121bbb7683b8056d2c5c51b3cac9f7b449b3c6975e7425db6e98b29"} Aug 03 23:34:30.262174 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:30.262152 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:30.263842 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:30.263817 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qhrx" event={"ID":"95547717-0f6c-42dc-9cc2-219289a032f7","Type":"ContainerStarted","Data":"d724f71a6348d9a787cc00cab026f96c09b9ec67ea616ae903a27a96b62292c6"} Aug 03 23:34:30.278803 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:30.278762 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" podStartSLOduration=21.744709912 podStartE2EDuration="24.278748291s" totalImagesPullingTime="2.534038379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:30.277783542 +0000 UTC m=+284.341039971" watchObservedRunningTime="2026-08-03 23:34:30.278748291 +0000 UTC m=+284.342004783" Aug 03 23:34:30.506905 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:30.506875 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-zp9bh" Aug 03 23:34:30.525369 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:30.524419 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5qhrx" podStartSLOduration=17.274786292 podStartE2EDuration="19.52440045s" totalImagesPullingTime="2.249614158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-03 23:34:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:30.29389515 +0000 UTC m=+284.357151581" watchObservedRunningTime="2026-08-03 23:34:30.52440045 +0000 UTC m=+284.587656879" Aug 03 23:34:30.935317 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:30.935225 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" podUID="d7aec628-af56-43fe-968c-81a79d5517f1" Aug 03 23:34:31.266837 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.266805 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:34:31.366838 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.366805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:34:31.367143 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.367117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:34:31.369451 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.369423 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2062046e-c1ef-42fb-b341-fab17c10c653-metrics-tls\") pod \"dns-default-w4tdk\" (UID: \"2062046e-c1ef-42fb-b341-fab17c10c653\") " pod="openshift-dns/dns-default-w4tdk" Aug 03 23:34:31.369543 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.369423 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"image-registry-575cb9dd4c-znjgv\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:34:31.558257 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.558189 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m6v2t\"" Aug 03 23:34:31.558257 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.558211 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rmmw5\"" Aug 03 23:34:31.565397 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.565380 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4tdk" Aug 03 23:34:31.565450 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.565417 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:34:31.698949 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.698917 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575cb9dd4c-znjgv"] Aug 03 23:34:31.702093 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:31.702060 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a44e778_2db8_449e_889e_07b17c8f8abe.slice/crio-c93a309ce376cd76fd6c47fc2ba835cf3c2bd08d1f2f5c2455d9030e907b24bb WatchSource:0}: Error finding container c93a309ce376cd76fd6c47fc2ba835cf3c2bd08d1f2f5c2455d9030e907b24bb: Status 404 returned error can't find the container with id c93a309ce376cd76fd6c47fc2ba835cf3c2bd08d1f2f5c2455d9030e907b24bb Aug 03 23:34:31.709755 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:31.709737 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w4tdk"] Aug 03 23:34:31.713365 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:31.713342 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2062046e_c1ef_42fb_b341_fab17c10c653.slice/crio-391c4079de864c4b3168e238b631e1d7e7503c6142bef8c9fb332df11e4605aa WatchSource:0}: Error finding container 391c4079de864c4b3168e238b631e1d7e7503c6142bef8c9fb332df11e4605aa: Status 404 returned error can't find the container with id 391c4079de864c4b3168e238b631e1d7e7503c6142bef8c9fb332df11e4605aa Aug 03 23:34:32.270765 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:32.270729 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4tdk" event={"ID":"2062046e-c1ef-42fb-b341-fab17c10c653","Type":"ContainerStarted","Data":"391c4079de864c4b3168e238b631e1d7e7503c6142bef8c9fb332df11e4605aa"} Aug 03 23:34:32.272184 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:32.272153 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" event={"ID":"8a44e778-2db8-449e-889e-07b17c8f8abe","Type":"ContainerStarted","Data":"c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c"} Aug 03 23:34:32.272370 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:32.272196 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" event={"ID":"8a44e778-2db8-449e-889e-07b17c8f8abe","Type":"ContainerStarted","Data":"c93a309ce376cd76fd6c47fc2ba835cf3c2bd08d1f2f5c2455d9030e907b24bb"} Aug 03 23:34:32.272370 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:32.272222 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:34:32.295357 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:32.295289 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" podStartSLOduration=286.295270781 podStartE2EDuration="4m46.295270781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:32.293074811 +0000 UTC m=+286.356331242" watchObservedRunningTime="2026-08-03 23:34:32.295270781 +0000 UTC m=+286.358527211" Aug 03 23:34:34.280886 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.280851 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4tdk" event={"ID":"2062046e-c1ef-42fb-b341-fab17c10c653","Type":"ContainerStarted","Data":"0abec1b804fa360ccbf3cb75850964361e5a75773836f60a1e0068e276ae6fd1"} Aug 03 23:34:34.281222 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.280929 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4tdk" event={"ID":"2062046e-c1ef-42fb-b341-fab17c10c653","Type":"ContainerStarted","Data":"242b6cac6be5a5a32f1ed5b2c5dbcc7e69cd23fbd3fe6febcc3025e954beb3db"} Aug 03 23:34:34.281222 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.280963 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w4tdk" Aug 03 23:34:34.297976 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.297930 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w4tdk" podStartSLOduration=251.315764032 podStartE2EDuration="4m13.297915351s" totalImagesPullingTime="1.982151319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:34.297287313 +0000 UTC m=+288.360543741" watchObservedRunningTime="2026-08-03 23:34:34.297915351 +0000 UTC m=+288.361171799" Aug 03 23:34:34.792290 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.792258 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:34:34.794658 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.794630 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d7aec628-af56-43fe-968c-81a79d5517f1-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9cxjw\" (UID: \"d7aec628-af56-43fe-968c-81a79d5517f1\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:34:34.870036 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.870009 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-svkcp\"" Aug 03 23:34:34.877940 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.877914 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" Aug 03 23:34:34.999400 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:34.999373 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw"] Aug 03 23:34:35.001860 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:35.001834 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7aec628_af56_43fe_968c_81a79d5517f1.slice/crio-fdf06fd936bfb5bb5e508cd952e0c67d15c60f2ba6e398c7d7714b9629f8deeb WatchSource:0}: Error finding container fdf06fd936bfb5bb5e508cd952e0c67d15c60f2ba6e398c7d7714b9629f8deeb: Status 404 returned error can't find the container with id fdf06fd936bfb5bb5e508cd952e0c67d15c60f2ba6e398c7d7714b9629f8deeb Aug 03 23:34:35.285239 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:35.285204 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" event={"ID":"d7aec628-af56-43fe-968c-81a79d5517f1","Type":"ContainerStarted","Data":"fdf06fd936bfb5bb5e508cd952e0c67d15c60f2ba6e398c7d7714b9629f8deeb"} Aug 03 23:34:36.290099 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:36.290061 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" event={"ID":"d7aec628-af56-43fe-968c-81a79d5517f1","Type":"ContainerStarted","Data":"24cfa37691d746b037cdf85026bb0d10ffee240752d58cfddd5bffb991630f11"} Aug 03 23:34:36.305139 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:36.305085 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9cxjw" podStartSLOduration=251.249208941 podStartE2EDuration="4m12.305069847s" totalImagesPullingTime="1.055860906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:36.304990216 +0000 UTC m=+290.368246644" watchObservedRunningTime="2026-08-03 23:34:36.305069847 +0000 UTC m=+290.368326275" Aug 03 23:34:40.416696 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:40.416655 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jsxr8" podUID="29374835-d52e-403b-9834-071b2f9bda81" Aug 03 23:34:44.287637 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.287609 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w4tdk" Aug 03 23:34:44.619593 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.619516 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp"] Aug 03 23:34:44.655887 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.655847 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz"] Aug 03 23:34:44.656041 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.655995 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.658236 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.658210 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nkd7x\"" Aug 03 23:34:44.658386 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.658282 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 03 23:34:44.659700 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.659420 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 03 23:34:44.659700 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.659452 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 03 23:34:44.659700 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.659456 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 03 23:34:44.659700 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.659627 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 03 23:34:44.669898 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.669879 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp"] Aug 03 23:34:44.669978 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.669915 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tlcrf"] Aug 03 23:34:44.670038 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.670001 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.672797 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.672776 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 03 23:34:44.672894 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.672817 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 03 23:34:44.673038 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.673015 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4brg4\"" Aug 03 23:34:44.673197 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.673020 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 03 23:34:44.683928 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.683909 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz"] Aug 03 23:34:44.684054 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.684037 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.686241 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.686213 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 03 23:34:44.687548 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.687379 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 03 23:34:44.687548 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.687406 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zn6kg\"" Aug 03 23:34:44.687548 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.687443 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 03 23:34:44.760632 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760603 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8408b47a-d8ca-4b65-99f8-344a836926cb-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.760632 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760635 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.760807 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760659 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.760807 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.760807 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760700 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-tls\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.760807 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760747 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.760807 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760784 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-textfile\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.760970 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760820 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwq49\" (UniqueName: \"kubernetes.io/projected/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-api-access-nwq49\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.760970 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.760835 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcw9h\" (UniqueName: \"kubernetes.io/projected/5d1b4b01-e938-4e2d-97b4-541bebc29b50-kube-api-access-mcw9h\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.761257 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.761231 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slhd\" (UniqueName: \"kubernetes.io/projected/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-kube-api-access-7slhd\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.761439 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.761410 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.761548 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.761535 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b4b01-e938-4e2d-97b4-541bebc29b50-metrics-client-ca\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.761667 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.761651 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8408b47a-d8ca-4b65-99f8-344a836926cb-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.761779 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.761764 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.761873 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.761858 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-sys\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.762013 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.761990 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.762068 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.762034 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-root\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.762113 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.762100 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-wtmp\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.762157 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.762134 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.863289 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863261 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.863476 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863476 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863345 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-tls\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863476 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863373 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.863476 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-textfile\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863476 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863449 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwq49\" (UniqueName: \"kubernetes.io/projected/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-api-access-nwq49\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.863476 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863468 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcw9h\" (UniqueName: \"kubernetes.io/projected/5d1b4b01-e938-4e2d-97b4-541bebc29b50-kube-api-access-mcw9h\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863494 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slhd\" (UniqueName: \"kubernetes.io/projected/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-kube-api-access-7slhd\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863517 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863541 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b4b01-e938-4e2d-97b4-541bebc29b50-metrics-client-ca\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863571 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8408b47a-d8ca-4b65-99f8-344a836926cb-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863596 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863619 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-sys\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863643 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-root\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863694 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-wtmp\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863729 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.863766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8408b47a-d8ca-4b65-99f8-344a836926cb-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.864238 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.863785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.864481 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.864457 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-wtmp\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.864822 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:44.864802 2556 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Aug 03 23:34:44.865082 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:44.864868 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-tls podName:8408b47a-d8ca-4b65-99f8-344a836926cb nodeName:}" failed. No retries permitted until 2026-08-03 23:34:45.364848506 +0000 UTC m=+299.428104916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-9krsz" (UID: "8408b47a-d8ca-4b65-99f8-344a836926cb") : secret "kube-state-metrics-tls" not found Aug 03 23:34:44.865423 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.865397 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8408b47a-d8ca-4b65-99f8-344a836926cb-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.865722 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.865702 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b4b01-e938-4e2d-97b4-541bebc29b50-metrics-client-ca\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.866044 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.866011 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-textfile\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.866465 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.866444 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-sys\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.866579 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.866469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.866667 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.866496 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.866762 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:44.866568 2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 03 23:34:44.866912 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:44.866900 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-tls podName:5d1b4b01-e938-4e2d-97b4-541bebc29b50 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:45.366881256 +0000 UTC m=+299.430137681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-tls") pod "node-exporter-tlcrf" (UID: "5d1b4b01-e938-4e2d-97b4-541bebc29b50") : secret "node-exporter-tls" not found Aug 03 23:34:44.867001 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.864703 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d1b4b01-e938-4e2d-97b4-541bebc29b50-root\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.867089 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:44.866617 2556 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Aug 03 23:34:44.867202 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:34:44.867191 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-tls podName:ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6 nodeName:}" failed. No retries permitted until 2026-08-03 23:34:45.367175875 +0000 UTC m=+299.430432282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-bgccp" (UID: "ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6") : secret "openshift-state-metrics-tls" not found Aug 03 23:34:44.867290 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.867238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8408b47a-d8ca-4b65-99f8-344a836926cb-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.868040 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.868018 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.868568 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.868530 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.869089 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.869032 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.870862 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.870796 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:44.877070 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.877028 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcw9h\" (UniqueName: \"kubernetes.io/projected/5d1b4b01-e938-4e2d-97b4-541bebc29b50-kube-api-access-mcw9h\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:44.880610 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.880589 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slhd\" (UniqueName: \"kubernetes.io/projected/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-kube-api-access-7slhd\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:44.880715 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:44.880696 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwq49\" (UniqueName: \"kubernetes.io/projected/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-api-access-nwq49\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:45.369817 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.369786 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:45.370261 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.369829 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-tls\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:45.370261 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.369850 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:45.372139 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.372119 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d1b4b01-e938-4e2d-97b4-541bebc29b50-node-exporter-tls\") pod \"node-exporter-tlcrf\" (UID: \"5d1b4b01-e938-4e2d-97b4-541bebc29b50\") " pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:45.372795 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.372772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8408b47a-d8ca-4b65-99f8-344a836926cb-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9krsz\" (UID: \"8408b47a-d8ca-4b65-99f8-344a836926cb\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:45.372861 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.372772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-bgccp\" (UID: \"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:45.566736 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.566690 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" Aug 03 23:34:45.579782 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.579757 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" Aug 03 23:34:45.593868 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.593836 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tlcrf" Aug 03 23:34:45.605968 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:45.605929 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1b4b01_e938_4e2d_97b4_541bebc29b50.slice/crio-6c1d0c59a4b67ea22149316da3920b8f82e111b7809b77ca8bffba64897714c0 WatchSource:0}: Error finding container 6c1d0c59a4b67ea22149316da3920b8f82e111b7809b77ca8bffba64897714c0: Status 404 returned error can't find the container with id 6c1d0c59a4b67ea22149316da3920b8f82e111b7809b77ca8bffba64897714c0 Aug 03 23:34:45.728776 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.728746 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp"] Aug 03 23:34:45.731772 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:45.731750 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd1d13e_ebf3_4f1a_a336_8d5bb734f8c6.slice/crio-25630ee112ad60e6a0f9768ed027993f4bf6a20cb0210ed22dd0792a8bdce643 WatchSource:0}: Error finding container 25630ee112ad60e6a0f9768ed027993f4bf6a20cb0210ed22dd0792a8bdce643: Status 404 returned error can't find the container with id 25630ee112ad60e6a0f9768ed027993f4bf6a20cb0210ed22dd0792a8bdce643 Aug 03 23:34:45.743283 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:45.743235 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz"] Aug 03 23:34:45.746288 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:45.746192 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8408b47a_d8ca_4b65_99f8_344a836926cb.slice/crio-ad13ce2064ad39b7d7cf281c7d61b8274f0518bdc54e4f81bf79a1d1c2b5813c WatchSource:0}: Error finding container ad13ce2064ad39b7d7cf281c7d61b8274f0518bdc54e4f81bf79a1d1c2b5813c: Status 404 returned error can't find the container with id ad13ce2064ad39b7d7cf281c7d61b8274f0518bdc54e4f81bf79a1d1c2b5813c Aug 03 23:34:46.326324 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:46.326266 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" event={"ID":"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6","Type":"ContainerStarted","Data":"b2bff46cb8399b9f271558fd805f6de7ccdf72c7781c8f098ecc9097d9b3be01"} Aug 03 23:34:46.326794 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:46.326761 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" event={"ID":"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6","Type":"ContainerStarted","Data":"00ac67d60327f75db0eaab555c5abd68dfe639770eab17d2bd00380657de5409"} Aug 03 23:34:46.326916 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:46.326809 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" event={"ID":"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6","Type":"ContainerStarted","Data":"25630ee112ad60e6a0f9768ed027993f4bf6a20cb0210ed22dd0792a8bdce643"} Aug 03 23:34:46.327655 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:46.327628 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" event={"ID":"8408b47a-d8ca-4b65-99f8-344a836926cb","Type":"ContainerStarted","Data":"ad13ce2064ad39b7d7cf281c7d61b8274f0518bdc54e4f81bf79a1d1c2b5813c"} Aug 03 23:34:46.329208 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:46.329178 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlcrf" event={"ID":"5d1b4b01-e938-4e2d-97b4-541bebc29b50","Type":"ContainerStarted","Data":"6c1d0c59a4b67ea22149316da3920b8f82e111b7809b77ca8bffba64897714c0"} Aug 03 23:34:46.526393 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:46.526342 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:34:46.528199 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:46.526906 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:34:46.539722 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:46.539703 2556 kubelet.go:1648] "Image garbage collection succeeded" Aug 03 23:34:47.337531 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:47.337166 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" event={"ID":"8408b47a-d8ca-4b65-99f8-344a836926cb","Type":"ContainerStarted","Data":"a4fb2becc7075b13e9fdca477f1447c424d4a8c8303873e33117bfaf040db96a"} Aug 03 23:34:47.337531 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:47.337290 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" event={"ID":"8408b47a-d8ca-4b65-99f8-344a836926cb","Type":"ContainerStarted","Data":"400a69684f19b5d49f4cae77e998569c5140cb0ed083aed16d27fe7c03c52737"} Aug 03 23:34:47.342051 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:47.341906 2556 generic.go:356] "Generic (PLEG): container finished" podID="5d1b4b01-e938-4e2d-97b4-541bebc29b50" containerID="83d32c44fd467c7e891f5e255b3ae8e7825fd5e59fe4ff1d52f27f0a42dc6316" exitCode=0 Aug 03 23:34:47.342051 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:47.341997 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlcrf" event={"ID":"5d1b4b01-e938-4e2d-97b4-541bebc29b50","Type":"ContainerDied","Data":"83d32c44fd467c7e891f5e255b3ae8e7825fd5e59fe4ff1d52f27f0a42dc6316"} Aug 03 23:34:47.346775 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:47.346689 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" event={"ID":"ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6","Type":"ContainerStarted","Data":"b7fca663d929e7d9ba1f7ea4577f20a8babe167d99124b3971a5f5eeef4d56b4"} Aug 03 23:34:47.379482 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:47.379440 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bgccp" podStartSLOduration=2.094072927 podStartE2EDuration="3.379425425s" totalImagesPullingTime="1.285352498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:47.377855519 +0000 UTC m=+301.441111948" watchObservedRunningTime="2026-08-03 23:34:47.379425425 +0000 UTC m=+301.442681853" Aug 03 23:34:48.352591 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.352554 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" event={"ID":"8408b47a-d8ca-4b65-99f8-344a836926cb","Type":"ContainerStarted","Data":"9922f0072fd0506c3825b2b60cba2678d5ca0ec6746b200a41e296dda42945f9"} Aug 03 23:34:48.354798 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.354762 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlcrf" event={"ID":"5d1b4b01-e938-4e2d-97b4-541bebc29b50","Type":"ContainerStarted","Data":"5d38f1f7f2d07018c3475e7fd1602e1729192327f209a8b4ac18ad504560d51c"} Aug 03 23:34:48.354931 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.354821 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlcrf" event={"ID":"5d1b4b01-e938-4e2d-97b4-541bebc29b50","Type":"ContainerStarted","Data":"03b989bbe9fbbe7798cc45af86f161eaf538f901e5141e1fe6eacb009ed261f7"} Aug 03 23:34:48.369798 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.369757 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9krsz" podStartSLOduration=2.976706246 podStartE2EDuration="4.369743073s" totalImagesPullingTime="1.393036827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:48.368639134 +0000 UTC m=+302.431895567" watchObservedRunningTime="2026-08-03 23:34:48.369743073 +0000 UTC m=+302.432999508" Aug 03 23:34:48.385828 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.385789 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tlcrf" podStartSLOduration=3.369445423 podStartE2EDuration="4.385771928s" totalImagesPullingTime="931.323781ms" totalInitContainerRuntime="85.002724ms" isStatefulPod=true podCreationTimestamp="2026-08-03 23:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:48.383865503 +0000 UTC m=+302.447121956" watchObservedRunningTime="2026-08-03 23:34:48.385771928 +0000 UTC m=+302.449028358" Aug 03 23:34:48.595911 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.595875 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9f667797f-zcbc9"] Aug 03 23:34:48.598836 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.598816 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.601193 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.601171 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8qqjc\"" Aug 03 23:34:48.601375 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.601358 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 03 23:34:48.601445 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.601435 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 03 23:34:48.601502 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.601451 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 03 23:34:48.601502 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.601451 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 03 23:34:48.601608 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.601512 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fmvueppj45mth\"" Aug 03 23:34:48.601608 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.601561 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 03 23:34:48.610269 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.610249 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9f667797f-zcbc9"] Aug 03 23:34:48.695581 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.695553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.695581 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.695583 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.695744 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.695600 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbpc2\" (UniqueName: \"kubernetes.io/projected/6dec7799-779d-4f15-9763-311dbb23ff21-kube-api-access-sbpc2\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.695744 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.695647 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-tls\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.695744 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.695666 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.695744 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.695713 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.695744 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.695732 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-grpc-tls\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.695906 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.695751 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dec7799-779d-4f15-9763-311dbb23ff21-metrics-client-ca\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.796622 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.796593 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.796731 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.796640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.796731 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.796684 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbpc2\" (UniqueName: \"kubernetes.io/projected/6dec7799-779d-4f15-9763-311dbb23ff21-kube-api-access-sbpc2\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.796820 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.796740 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-tls\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.796820 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.796769 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.796916 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.796845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.796916 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.796868 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-grpc-tls\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.796916 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.796896 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dec7799-779d-4f15-9763-311dbb23ff21-metrics-client-ca\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.797835 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.797807 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dec7799-779d-4f15-9763-311dbb23ff21-metrics-client-ca\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.799415 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.799392 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.799490 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.799435 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.799936 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.799907 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-tls\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.800038 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.800015 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.800139 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.800120 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.800200 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.800186 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dec7799-779d-4f15-9763-311dbb23ff21-secret-grpc-tls\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.804840 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.804822 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbpc2\" (UniqueName: \"kubernetes.io/projected/6dec7799-779d-4f15-9763-311dbb23ff21-kube-api-access-sbpc2\") pod \"thanos-querier-9f667797f-zcbc9\" (UID: \"6dec7799-779d-4f15-9763-311dbb23ff21\") " pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:48.908178 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:48.908121 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:49.039171 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.039145 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9f667797f-zcbc9"] Aug 03 23:34:49.041509 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:49.041482 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dec7799_779d_4f15_9763_311dbb23ff21.slice/crio-cfa12816b4e598c033ff046617a2ae8ee16a26e567e27ebdf4f2422c58a9718d WatchSource:0}: Error finding container cfa12816b4e598c033ff046617a2ae8ee16a26e567e27ebdf4f2422c58a9718d: Status 404 returned error can't find the container with id cfa12816b4e598c033ff046617a2ae8ee16a26e567e27ebdf4f2422c58a9718d Aug 03 23:34:49.043826 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.043810 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 03 23:34:49.360723 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.360688 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" event={"ID":"6dec7799-779d-4f15-9763-311dbb23ff21","Type":"ContainerStarted","Data":"cfa12816b4e598c033ff046617a2ae8ee16a26e567e27ebdf4f2422c58a9718d"} Aug 03 23:34:49.395162 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.395129 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j"] Aug 03 23:34:49.398250 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.398232 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:49.403544 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.402749 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bvbwh\"" Aug 03 23:34:49.403544 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.402997 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 03 23:34:49.406085 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.406059 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j"] Aug 03 23:34:49.504600 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.504576 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrzpk\" (UniqueName: \"kubernetes.io/projected/86336890-d32f-4d81-9b89-dcd8adeacb93-kube-api-access-vrzpk\") pod \"monitoring-plugin-6dffbbd67-vgn9j\" (UID: \"86336890-d32f-4d81-9b89-dcd8adeacb93\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:49.505009 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.504762 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/86336890-d32f-4d81-9b89-dcd8adeacb93-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vgn9j\" (UID: \"86336890-d32f-4d81-9b89-dcd8adeacb93\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:49.606167 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.606136 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/86336890-d32f-4d81-9b89-dcd8adeacb93-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vgn9j\" (UID: \"86336890-d32f-4d81-9b89-dcd8adeacb93\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:49.606292 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.606194 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrzpk\" (UniqueName: \"kubernetes.io/projected/86336890-d32f-4d81-9b89-dcd8adeacb93-kube-api-access-vrzpk\") pod \"monitoring-plugin-6dffbbd67-vgn9j\" (UID: \"86336890-d32f-4d81-9b89-dcd8adeacb93\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:49.608628 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.608599 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/86336890-d32f-4d81-9b89-dcd8adeacb93-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-vgn9j\" (UID: \"86336890-d32f-4d81-9b89-dcd8adeacb93\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:49.614356 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.614290 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrzpk\" (UniqueName: \"kubernetes.io/projected/86336890-d32f-4d81-9b89-dcd8adeacb93-kube-api-access-vrzpk\") pod \"monitoring-plugin-6dffbbd67-vgn9j\" (UID: \"86336890-d32f-4d81-9b89-dcd8adeacb93\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:49.712917 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.712892 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:49.834018 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:49.833987 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j"] Aug 03 23:34:49.837502 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:49.837471 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86336890_d32f_4d81_9b89_dcd8adeacb93.slice/crio-38bf10b96e508b2967b35e0c8bf576050c53e71e60e477861526ae002575486b WatchSource:0}: Error finding container 38bf10b96e508b2967b35e0c8bf576050c53e71e60e477861526ae002575486b: Status 404 returned error can't find the container with id 38bf10b96e508b2967b35e0c8bf576050c53e71e60e477861526ae002575486b Aug 03 23:34:50.366239 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:50.366205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" event={"ID":"86336890-d32f-4d81-9b89-dcd8adeacb93","Type":"ContainerStarted","Data":"38bf10b96e508b2967b35e0c8bf576050c53e71e60e477861526ae002575486b"} Aug 03 23:34:51.385796 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.385761 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-674cccc6cb-xml87"] Aug 03 23:34:51.388495 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.388464 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.391129 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.391105 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 03 23:34:51.392455 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.392430 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 03 23:34:51.392455 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.392430 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 03 23:34:51.392703 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.392681 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fw459\"" Aug 03 23:34:51.392792 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.392773 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 03 23:34:51.392839 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.392798 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 03 23:34:51.392875 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.392849 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 03 23:34:51.393122 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.393105 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 03 23:34:51.399240 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.399217 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 03 23:34:51.399668 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.399646 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-674cccc6cb-xml87"] Aug 03 23:34:51.523743 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.523715 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-oauth-serving-cert\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.523921 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.523782 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-serving-cert\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.523921 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.523808 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-trusted-ca-bundle\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.523921 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.523833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-oauth-config\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.524130 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.523925 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-console-config\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.524130 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.523954 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-service-ca\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.524130 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.523996 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf9p5\" (UniqueName: \"kubernetes.io/projected/14230787-3f8e-4619-bfc2-9598fbf333f4-kube-api-access-gf9p5\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.571084 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.571057 2556 patch_prober.go:28] interesting pod/image-registry-575cb9dd4c-znjgv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 03 23:34:51.571190 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.571105 2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" podUID="8a44e778-2db8-449e-889e-07b17c8f8abe" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 03 23:34:51.624634 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.624611 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-oauth-config\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.624728 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.624712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-console-config\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.624790 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.624746 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-service-ca\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.624889 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.624796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf9p5\" (UniqueName: \"kubernetes.io/projected/14230787-3f8e-4619-bfc2-9598fbf333f4-kube-api-access-gf9p5\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.624889 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.624837 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-oauth-serving-cert\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.624984 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.624889 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-serving-cert\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.624984 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.624916 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-trusted-ca-bundle\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.625838 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.625784 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-service-ca\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.625947 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.625862 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-console-config\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.625947 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.625922 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-oauth-serving-cert\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.626070 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.626048 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-trusted-ca-bundle\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.628919 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.628856 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-oauth-config\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.628919 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.628889 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-serving-cert\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.633875 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.633849 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf9p5\" (UniqueName: \"kubernetes.io/projected/14230787-3f8e-4619-bfc2-9598fbf333f4-kube-api-access-gf9p5\") pod \"console-674cccc6cb-xml87\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.701057 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.701036 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:34:51.831671 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:51.831638 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-674cccc6cb-xml87"] Aug 03 23:34:51.835619 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:34:51.835593 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14230787_3f8e_4619_bfc2_9598fbf333f4.slice/crio-911e4d38eff3deb7b114eda2c111bd810a36c8d939da98f1f503ee37c11dd497 WatchSource:0}: Error finding container 911e4d38eff3deb7b114eda2c111bd810a36c8d939da98f1f503ee37c11dd497: Status 404 returned error can't find the container with id 911e4d38eff3deb7b114eda2c111bd810a36c8d939da98f1f503ee37c11dd497 Aug 03 23:34:52.379426 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:52.379389 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" event={"ID":"86336890-d32f-4d81-9b89-dcd8adeacb93","Type":"ContainerStarted","Data":"cb41c4103a696ca17d09f538466591f1d2272c5392fca24ba0acd86c6ed4c94a"} Aug 03 23:34:52.379611 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:52.379597 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:52.382603 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:52.382571 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" event={"ID":"6dec7799-779d-4f15-9763-311dbb23ff21","Type":"ContainerStarted","Data":"6994419e3340f7b9ccc8a756fa3b9f49fdea482e00084b24777cbeff32dc87f0"} Aug 03 23:34:52.382738 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:52.382645 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" event={"ID":"6dec7799-779d-4f15-9763-311dbb23ff21","Type":"ContainerStarted","Data":"c2526b38e1908369ecba84098e4a45b4cf6cda63f0aaf7e2a659d9f46b435a5b"} Aug 03 23:34:52.382738 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:52.382687 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" event={"ID":"6dec7799-779d-4f15-9763-311dbb23ff21","Type":"ContainerStarted","Data":"170824393cb98153cbe92d69ed99219f0b099a75250bbf4c954baa0572e8a5e6"} Aug 03 23:34:52.384011 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:52.383987 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674cccc6cb-xml87" event={"ID":"14230787-3f8e-4619-bfc2-9598fbf333f4","Type":"ContainerStarted","Data":"911e4d38eff3deb7b114eda2c111bd810a36c8d939da98f1f503ee37c11dd497"} Aug 03 23:34:52.386855 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:52.386836 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" Aug 03 23:34:52.393995 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:52.393960 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-vgn9j" podStartSLOduration=1.6891220009999999 podStartE2EDuration="3.393946944s" totalImagesPullingTime="1.704824943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:52.393415291 +0000 UTC m=+306.456671723" watchObservedRunningTime="2026-08-03 23:34:52.393946944 +0000 UTC m=+306.457203374" Aug 03 23:34:53.281513 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:53.281482 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:34:53.392347 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:53.392282 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" event={"ID":"6dec7799-779d-4f15-9763-311dbb23ff21","Type":"ContainerStarted","Data":"b999421df6a87c7739d808486707e41fbcd36e0b2378062cde380866c0807e39"} Aug 03 23:34:53.392782 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:53.392386 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" event={"ID":"6dec7799-779d-4f15-9763-311dbb23ff21","Type":"ContainerStarted","Data":"f1b7871f988e2d4f65f4208ce763587753cdcc66659077aff1e9249e9966b00f"} Aug 03 23:34:53.392782 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:53.392433 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" event={"ID":"6dec7799-779d-4f15-9763-311dbb23ff21","Type":"ContainerStarted","Data":"dff16f07e33518a83de0ee832bf9d73464ebe965d206d920eda3c7ce70b1420f"} Aug 03 23:34:53.412175 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:53.412130 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" podStartSLOduration=1.9445382439999999 podStartE2EDuration="5.41211372s" totalImagesPullingTime="3.467575476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:53.411138102 +0000 UTC m=+307.474394531" watchObservedRunningTime="2026-08-03 23:34:53.41211372 +0000 UTC m=+307.475370148" Aug 03 23:34:54.396967 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:54.396937 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:34:55.401901 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:55.401864 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674cccc6cb-xml87" event={"ID":"14230787-3f8e-4619-bfc2-9598fbf333f4","Type":"ContainerStarted","Data":"a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7"} Aug 03 23:34:55.415204 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:55.415184 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:34:55.418344 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:55.418279 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-674cccc6cb-xml87" podStartSLOduration=1.8344222719999999 podStartE2EDuration="4.418266431s" totalImagesPullingTime="2.583844159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:34:55.416908923 +0000 UTC m=+309.480165352" watchObservedRunningTime="2026-08-03 23:34:55.418266431 +0000 UTC m=+309.481522859" Aug 03 23:34:57.602879 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:34:57.602849 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-575cb9dd4c-znjgv"] Aug 03 23:35:00.407812 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:00.407782 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9f667797f-zcbc9" Aug 03 23:35:01.701480 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:01.701442 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:35:01.701864 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:01.701516 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:35:01.706094 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:01.706073 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:35:02.435090 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:02.435064 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:35:22.621876 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.621826 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" podUID="8a44e778-2db8-449e-889e-07b17c8f8abe" containerName="registry" containerID="cri-o://c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c" gracePeriod=30 Aug 03 23:35:22.862060 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.862035 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:35:22.980566 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980534 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-certificates\") pod \"8a44e778-2db8-449e-889e-07b17c8f8abe\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " Aug 03 23:35:22.980734 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980581 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") pod \"8a44e778-2db8-449e-889e-07b17c8f8abe\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " Aug 03 23:35:22.980734 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980610 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-bound-sa-token\") pod \"8a44e778-2db8-449e-889e-07b17c8f8abe\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " Aug 03 23:35:22.980734 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980653 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-image-registry-private-configuration\") pod \"8a44e778-2db8-449e-889e-07b17c8f8abe\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " Aug 03 23:35:22.980734 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980699 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a44e778-2db8-449e-889e-07b17c8f8abe-ca-trust-extracted\") pod \"8a44e778-2db8-449e-889e-07b17c8f8abe\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " Aug 03 23:35:22.980734 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980728 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-installation-pull-secrets\") pod \"8a44e778-2db8-449e-889e-07b17c8f8abe\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " Aug 03 23:35:22.980981 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980778 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-trusted-ca\") pod \"8a44e778-2db8-449e-889e-07b17c8f8abe\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " Aug 03 23:35:22.980981 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980807 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzk98\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-kube-api-access-rzk98\") pod \"8a44e778-2db8-449e-889e-07b17c8f8abe\" (UID: \"8a44e778-2db8-449e-889e-07b17c8f8abe\") " Aug 03 23:35:22.980981 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.980932 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8a44e778-2db8-449e-889e-07b17c8f8abe" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:35:22.981237 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.981214 2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-certificates\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:35:22.981601 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.981394 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8a44e778-2db8-449e-889e-07b17c8f8abe" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:35:22.983338 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.983297 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-kube-api-access-rzk98" (OuterVolumeSpecName: "kube-api-access-rzk98") pod "8a44e778-2db8-449e-889e-07b17c8f8abe" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe"). InnerVolumeSpecName "kube-api-access-rzk98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:35:22.983442 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.983348 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8a44e778-2db8-449e-889e-07b17c8f8abe" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 03 23:35:22.983442 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.983392 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8a44e778-2db8-449e-889e-07b17c8f8abe" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:35:22.983550 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.983482 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8a44e778-2db8-449e-889e-07b17c8f8abe" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:35:22.983803 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.983775 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8a44e778-2db8-449e-889e-07b17c8f8abe" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 03 23:35:22.989165 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:22.989141 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a44e778-2db8-449e-889e-07b17c8f8abe-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8a44e778-2db8-449e-889e-07b17c8f8abe" (UID: "8a44e778-2db8-449e-889e-07b17c8f8abe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:35:23.082448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.082392 2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a44e778-2db8-449e-889e-07b17c8f8abe-ca-trust-extracted\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:35:23.082448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.082440 2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-installation-pull-secrets\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:35:23.082448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.082451 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a44e778-2db8-449e-889e-07b17c8f8abe-trusted-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:35:23.082448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.082459 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzk98\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-kube-api-access-rzk98\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:35:23.082448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.082468 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-registry-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:35:23.082740 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.082477 2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a44e778-2db8-449e-889e-07b17c8f8abe-bound-sa-token\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:35:23.082740 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.082485 2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a44e778-2db8-449e-889e-07b17c8f8abe-image-registry-private-configuration\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:35:23.511150 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.511115 2556 generic.go:356] "Generic (PLEG): container finished" podID="8a44e778-2db8-449e-889e-07b17c8f8abe" containerID="c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c" exitCode=0 Aug 03 23:35:23.511361 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.511179 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" event={"ID":"8a44e778-2db8-449e-889e-07b17c8f8abe","Type":"ContainerDied","Data":"c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c"} Aug 03 23:35:23.511361 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.511187 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" Aug 03 23:35:23.511361 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.511218 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575cb9dd4c-znjgv" event={"ID":"8a44e778-2db8-449e-889e-07b17c8f8abe","Type":"ContainerDied","Data":"c93a309ce376cd76fd6c47fc2ba835cf3c2bd08d1f2f5c2455d9030e907b24bb"} Aug 03 23:35:23.511361 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.511238 2556 scope.go:117] "RemoveContainer" containerID="c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c" Aug 03 23:35:23.520834 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.520814 2556 scope.go:117] "RemoveContainer" containerID="c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c" Aug 03 23:35:23.521094 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:35:23.521075 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c\": container with ID starting with c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c not found: ID does not exist" containerID="c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c" Aug 03 23:35:23.521152 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.521101 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c"} err="failed to get container status \"c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c\": rpc error: code = NotFound desc = could not find container \"c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c\": container with ID starting with c6ca6badd9216c6e9fab4c67cf2ddd28ab00544d69b37b2cd14ea3c3dd83c53c not found: ID does not exist" Aug 03 23:35:23.534268 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.534245 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-575cb9dd4c-znjgv"] Aug 03 23:35:23.537936 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:23.537916 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-575cb9dd4c-znjgv"] Aug 03 23:35:24.420478 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:24.420443 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a44e778-2db8-449e-889e-07b17c8f8abe" path="/var/lib/kubelet/pods/8a44e778-2db8-449e-889e-07b17c8f8abe/volumes" Aug 03 23:35:30.538672 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:30.538634 2556 generic.go:356] "Generic (PLEG): container finished" podID="cb55347b-ee57-4828-a76c-2bc48d562e0e" containerID="534a93636f976f6c6df4e83d57215d624e783c13a2af22e0839ed573957db609" exitCode=0 Aug 03 23:35:30.539175 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:30.538689 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" event={"ID":"cb55347b-ee57-4828-a76c-2bc48d562e0e","Type":"ContainerDied","Data":"534a93636f976f6c6df4e83d57215d624e783c13a2af22e0839ed573957db609"} Aug 03 23:35:30.539175 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:30.539081 2556 scope.go:117] "RemoveContainer" containerID="534a93636f976f6c6df4e83d57215d624e783c13a2af22e0839ed573957db609" Aug 03 23:35:31.544124 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:35:31.544093 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-d98z5" event={"ID":"cb55347b-ee57-4828-a76c-2bc48d562e0e","Type":"ContainerStarted","Data":"4e632addafcbf7c85ec0bdabddb8bd20161ad674bb559b51be8a59b4fa6813af"} Aug 03 23:36:00.295430 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:00.295390 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:36:00.297838 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:00.297808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29374835-d52e-403b-9834-071b2f9bda81-metrics-certs\") pod \"network-metrics-daemon-jsxr8\" (UID: \"29374835-d52e-403b-9834-071b2f9bda81\") " pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:36:00.518946 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:00.518914 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6szp\"" Aug 03 23:36:00.526983 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:00.526945 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsxr8" Aug 03 23:36:00.657325 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:00.657280 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jsxr8"] Aug 03 23:36:00.660221 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:36:00.660188 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29374835_d52e_403b_9834_071b2f9bda81.slice/crio-0373688b53a78347bd89320b0cfab53b835a2207e182a41a62edcf82bfa08ff6 WatchSource:0}: Error finding container 0373688b53a78347bd89320b0cfab53b835a2207e182a41a62edcf82bfa08ff6: Status 404 returned error can't find the container with id 0373688b53a78347bd89320b0cfab53b835a2207e182a41a62edcf82bfa08ff6 Aug 03 23:36:01.663050 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:01.663011 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsxr8" event={"ID":"29374835-d52e-403b-9834-071b2f9bda81","Type":"ContainerStarted","Data":"0373688b53a78347bd89320b0cfab53b835a2207e182a41a62edcf82bfa08ff6"} Aug 03 23:36:02.673808 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:02.673721 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsxr8" event={"ID":"29374835-d52e-403b-9834-071b2f9bda81","Type":"ContainerStarted","Data":"0cfe1131f878502cfe91475707f1c7aedc106d5635681bcb9922d5b4ed06ad21"} Aug 03 23:36:02.673808 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:02.673806 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsxr8" event={"ID":"29374835-d52e-403b-9834-071b2f9bda81","Type":"ContainerStarted","Data":"e13e15850b07a70ffe9aa8ffe827aba6d9db8820c35ce0af6a320ec056874412"} Aug 03 23:36:02.690242 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:02.690181 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jsxr8" podStartSLOduration=375.093965467 podStartE2EDuration="6m16.690165674s" totalImagesPullingTime="1.596200207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:36:02.688589887 +0000 UTC m=+376.751846316" watchObservedRunningTime="2026-08-03 23:36:02.690165674 +0000 UTC m=+376.753422103" Aug 03 23:36:12.172205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.172163 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fffdb9589-xx4db"] Aug 03 23:36:12.172745 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.172644 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a44e778-2db8-449e-889e-07b17c8f8abe" containerName="registry" Aug 03 23:36:12.172745 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.172664 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a44e778-2db8-449e-889e-07b17c8f8abe" containerName="registry" Aug 03 23:36:12.172856 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.172763 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a44e778-2db8-449e-889e-07b17c8f8abe" containerName="registry" Aug 03 23:36:12.175032 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.175008 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.186158 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.186133 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fffdb9589-xx4db"] Aug 03 23:36:12.291179 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.291135 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-console-config\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.291396 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.291184 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-trusted-ca-bundle\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.291396 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.291211 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-oauth-serving-cert\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.291396 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.291272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-service-ca\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.291396 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.291383 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-serving-cert\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.291569 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.291416 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbl4\" (UniqueName: \"kubernetes.io/projected/569c42c8-855a-4208-982a-b238ddaf3ee9-kube-api-access-ljbl4\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.291569 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.291437 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-oauth-config\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.392439 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.392401 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-serving-cert\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.392439 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.392442 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbl4\" (UniqueName: \"kubernetes.io/projected/569c42c8-855a-4208-982a-b238ddaf3ee9-kube-api-access-ljbl4\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.392681 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.392468 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-oauth-config\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.392681 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.392517 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-console-config\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.392681 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.392546 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-trusted-ca-bundle\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.392681 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.392568 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-oauth-serving-cert\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.392681 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.392605 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-service-ca\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.393293 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.393267 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-service-ca\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.393437 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.393410 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-console-config\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.393503 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.393486 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-trusted-ca-bundle\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.393671 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.393648 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-oauth-serving-cert\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.394999 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.394978 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-oauth-config\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.395119 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.395098 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-serving-cert\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.399859 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.399837 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbl4\" (UniqueName: \"kubernetes.io/projected/569c42c8-855a-4208-982a-b238ddaf3ee9-kube-api-access-ljbl4\") pod \"console-5fffdb9589-xx4db\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.487024 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.486997 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:12.623815 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.623778 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fffdb9589-xx4db"] Aug 03 23:36:12.626638 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:36:12.626602 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569c42c8_855a_4208_982a_b238ddaf3ee9.slice/crio-9b59b5db5347f5bea85dab3527cc0d2d707733580a4eb5e702d6166a1c20dd29 WatchSource:0}: Error finding container 9b59b5db5347f5bea85dab3527cc0d2d707733580a4eb5e702d6166a1c20dd29: Status 404 returned error can't find the container with id 9b59b5db5347f5bea85dab3527cc0d2d707733580a4eb5e702d6166a1c20dd29 Aug 03 23:36:12.713962 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.713925 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fffdb9589-xx4db" event={"ID":"569c42c8-855a-4208-982a-b238ddaf3ee9","Type":"ContainerStarted","Data":"f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f"} Aug 03 23:36:12.714120 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.713977 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fffdb9589-xx4db" event={"ID":"569c42c8-855a-4208-982a-b238ddaf3ee9","Type":"ContainerStarted","Data":"9b59b5db5347f5bea85dab3527cc0d2d707733580a4eb5e702d6166a1c20dd29"} Aug 03 23:36:12.730471 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:12.730414 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fffdb9589-xx4db" podStartSLOduration=0.730395246 podStartE2EDuration="730.395246ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:36:12.729268394 +0000 UTC m=+386.792524826" watchObservedRunningTime="2026-08-03 23:36:12.730395246 +0000 UTC m=+386.793652023" Aug 03 23:36:22.487290 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:22.487257 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:22.487825 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:22.487333 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:22.492040 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:22.492020 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:22.755696 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:22.755622 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:36:22.800808 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:22.800779 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-674cccc6cb-xml87"] Aug 03 23:36:47.822220 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:47.822178 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-674cccc6cb-xml87" podUID="14230787-3f8e-4619-bfc2-9598fbf333f4" containerName="console" containerID="cri-o://a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7" gracePeriod=15 Aug 03 23:36:48.056364 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.056341 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:36:48.190125 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190040 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf9p5\" (UniqueName: \"kubernetes.io/projected/14230787-3f8e-4619-bfc2-9598fbf333f4-kube-api-access-gf9p5\") pod \"14230787-3f8e-4619-bfc2-9598fbf333f4\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " Aug 03 23:36:48.190125 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190103 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-oauth-serving-cert\") pod \"14230787-3f8e-4619-bfc2-9598fbf333f4\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " Aug 03 23:36:48.190125 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190122 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-oauth-config\") pod \"14230787-3f8e-4619-bfc2-9598fbf333f4\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " Aug 03 23:36:48.190363 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190141 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-trusted-ca-bundle\") pod \"14230787-3f8e-4619-bfc2-9598fbf333f4\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " Aug 03 23:36:48.190363 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190256 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-serving-cert\") pod \"14230787-3f8e-4619-bfc2-9598fbf333f4\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " Aug 03 23:36:48.190363 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190357 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-service-ca\") pod \"14230787-3f8e-4619-bfc2-9598fbf333f4\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " Aug 03 23:36:48.190510 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190397 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-console-config\") pod \"14230787-3f8e-4619-bfc2-9598fbf333f4\" (UID: \"14230787-3f8e-4619-bfc2-9598fbf333f4\") " Aug 03 23:36:48.190678 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190647 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "14230787-3f8e-4619-bfc2-9598fbf333f4" (UID: "14230787-3f8e-4619-bfc2-9598fbf333f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:36:48.190678 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190655 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "14230787-3f8e-4619-bfc2-9598fbf333f4" (UID: "14230787-3f8e-4619-bfc2-9598fbf333f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:36:48.190878 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190849 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "14230787-3f8e-4619-bfc2-9598fbf333f4" (UID: "14230787-3f8e-4619-bfc2-9598fbf333f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:36:48.190931 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.190882 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-console-config" (OuterVolumeSpecName: "console-config") pod "14230787-3f8e-4619-bfc2-9598fbf333f4" (UID: "14230787-3f8e-4619-bfc2-9598fbf333f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:36:48.192606 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.192582 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "14230787-3f8e-4619-bfc2-9598fbf333f4" (UID: "14230787-3f8e-4619-bfc2-9598fbf333f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 03 23:36:48.192870 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.192851 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14230787-3f8e-4619-bfc2-9598fbf333f4-kube-api-access-gf9p5" (OuterVolumeSpecName: "kube-api-access-gf9p5") pod "14230787-3f8e-4619-bfc2-9598fbf333f4" (UID: "14230787-3f8e-4619-bfc2-9598fbf333f4"). InnerVolumeSpecName "kube-api-access-gf9p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:36:48.192920 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.192853 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "14230787-3f8e-4619-bfc2-9598fbf333f4" (UID: "14230787-3f8e-4619-bfc2-9598fbf333f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 03 23:36:48.290967 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.290935 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf9p5\" (UniqueName: \"kubernetes.io/projected/14230787-3f8e-4619-bfc2-9598fbf333f4-kube-api-access-gf9p5\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:36:48.290967 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.290961 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-oauth-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:36:48.290967 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.290971 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-oauth-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:36:48.291162 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.290979 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-trusted-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:36:48.291162 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.290987 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14230787-3f8e-4619-bfc2-9598fbf333f4-console-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:36:48.291162 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.290995 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:36:48.291162 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.291003 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14230787-3f8e-4619-bfc2-9598fbf333f4-console-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:36:48.858889 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.858855 2556 generic.go:356] "Generic (PLEG): container finished" podID="14230787-3f8e-4619-bfc2-9598fbf333f4" containerID="a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7" exitCode=0 Aug 03 23:36:48.859294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.858897 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674cccc6cb-xml87" event={"ID":"14230787-3f8e-4619-bfc2-9598fbf333f4","Type":"ContainerDied","Data":"a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7"} Aug 03 23:36:48.859294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.858912 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674cccc6cb-xml87" Aug 03 23:36:48.859294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.858964 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674cccc6cb-xml87" event={"ID":"14230787-3f8e-4619-bfc2-9598fbf333f4","Type":"ContainerDied","Data":"911e4d38eff3deb7b114eda2c111bd810a36c8d939da98f1f503ee37c11dd497"} Aug 03 23:36:48.859294 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.858985 2556 scope.go:117] "RemoveContainer" containerID="a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7" Aug 03 23:36:48.867239 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.867221 2556 scope.go:117] "RemoveContainer" containerID="a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7" Aug 03 23:36:48.867540 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:36:48.867515 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7\": container with ID starting with a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7 not found: ID does not exist" containerID="a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7" Aug 03 23:36:48.867640 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.867545 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7"} err="failed to get container status \"a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7\": rpc error: code = NotFound desc = could not find container \"a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7\": container with ID starting with a53aeea1c8176db1543359e55347b441eca58ecc17cdaec6d23f0b84863b51b7 not found: ID does not exist" Aug 03 23:36:48.875932 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.875911 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-674cccc6cb-xml87"] Aug 03 23:36:48.879581 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:48.879561 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-674cccc6cb-xml87"] Aug 03 23:36:50.420086 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:36:50.420053 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14230787-3f8e-4619-bfc2-9598fbf333f4" path="/var/lib/kubelet/pods/14230787-3f8e-4619-bfc2-9598fbf333f4/volumes" Aug 03 23:37:22.499681 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.499651 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c4c79b898-9krwv"] Aug 03 23:37:22.500139 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.499946 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14230787-3f8e-4619-bfc2-9598fbf333f4" containerName="console" Aug 03 23:37:22.500139 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.499958 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14230787-3f8e-4619-bfc2-9598fbf333f4" containerName="console" Aug 03 23:37:22.500139 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.500025 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14230787-3f8e-4619-bfc2-9598fbf333f4" containerName="console" Aug 03 23:37:22.503097 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.503064 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.514644 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.514615 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4c79b898-9krwv"] Aug 03 23:37:22.542935 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.542895 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-serving-cert\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.543083 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.542949 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-trusted-ca-bundle\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.543083 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.542987 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-oauth-config\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.543083 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.543073 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nt7\" (UniqueName: \"kubernetes.io/projected/a913c316-f6ec-427f-aca8-890f9318e20f-kube-api-access-c6nt7\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.543226 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.543114 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-service-ca\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.543226 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.543167 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-oauth-serving-cert\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.543226 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.543195 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-console-config\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.643924 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.643885 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-serving-cert\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.643924 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.643926 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-trusted-ca-bundle\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.644155 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.643957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-oauth-config\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.644155 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.644005 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nt7\" (UniqueName: \"kubernetes.io/projected/a913c316-f6ec-427f-aca8-890f9318e20f-kube-api-access-c6nt7\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.644155 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.644034 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-service-ca\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.644155 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.644081 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-oauth-serving-cert\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.644155 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.644106 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-console-config\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.644871 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.644842 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-service-ca\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.644976 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.644885 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-oauth-serving-cert\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.644976 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.644894 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-trusted-ca-bundle\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.645048 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.645023 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-console-config\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.646573 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.646548 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-oauth-config\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.646745 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.646725 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-serving-cert\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.651890 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.651865 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nt7\" (UniqueName: \"kubernetes.io/projected/a913c316-f6ec-427f-aca8-890f9318e20f-kube-api-access-c6nt7\") pod \"console-7c4c79b898-9krwv\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.815405 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.815292 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:22.954721 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.954695 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4c79b898-9krwv"] Aug 03 23:37:22.956994 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:37:22.956960 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda913c316_f6ec_427f_aca8_890f9318e20f.slice/crio-3e5256a1f239656d76682dd336355ae83cbd09f8716a6d0e4a7e4f6b30c8203e WatchSource:0}: Error finding container 3e5256a1f239656d76682dd336355ae83cbd09f8716a6d0e4a7e4f6b30c8203e: Status 404 returned error can't find the container with id 3e5256a1f239656d76682dd336355ae83cbd09f8716a6d0e4a7e4f6b30c8203e Aug 03 23:37:22.996692 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:22.996662 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4c79b898-9krwv" event={"ID":"a913c316-f6ec-427f-aca8-890f9318e20f","Type":"ContainerStarted","Data":"3e5256a1f239656d76682dd336355ae83cbd09f8716a6d0e4a7e4f6b30c8203e"} Aug 03 23:37:24.001988 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:24.001956 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4c79b898-9krwv" event={"ID":"a913c316-f6ec-427f-aca8-890f9318e20f","Type":"ContainerStarted","Data":"b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d"} Aug 03 23:37:24.019413 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:24.019371 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c4c79b898-9krwv" podStartSLOduration=2.019358157 podStartE2EDuration="2.019358157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:37:24.017770826 +0000 UTC m=+458.081027258" watchObservedRunningTime="2026-08-03 23:37:24.019358157 +0000 UTC m=+458.082614587" Aug 03 23:37:32.815745 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:32.815705 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:32.816128 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:32.815782 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:32.820405 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:32.820382 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:33.043129 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:33.043101 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:37:33.087780 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:33.087701 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fffdb9589-xx4db"] Aug 03 23:37:58.114365 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.114292 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fffdb9589-xx4db" podUID="569c42c8-855a-4208-982a-b238ddaf3ee9" containerName="console" containerID="cri-o://f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f" gracePeriod=15 Aug 03 23:37:58.355263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.355235 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:37:58.431144 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431042 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-service-ca\") pod \"569c42c8-855a-4208-982a-b238ddaf3ee9\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " Aug 03 23:37:58.431144 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431115 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljbl4\" (UniqueName: \"kubernetes.io/projected/569c42c8-855a-4208-982a-b238ddaf3ee9-kube-api-access-ljbl4\") pod \"569c42c8-855a-4208-982a-b238ddaf3ee9\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " Aug 03 23:37:58.431402 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431154 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-oauth-serving-cert\") pod \"569c42c8-855a-4208-982a-b238ddaf3ee9\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " Aug 03 23:37:58.431402 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431185 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-serving-cert\") pod \"569c42c8-855a-4208-982a-b238ddaf3ee9\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " Aug 03 23:37:58.431402 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431211 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-console-config\") pod \"569c42c8-855a-4208-982a-b238ddaf3ee9\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " Aug 03 23:37:58.431559 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431392 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-trusted-ca-bundle\") pod \"569c42c8-855a-4208-982a-b238ddaf3ee9\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " Aug 03 23:37:58.431559 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431449 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-oauth-config\") pod \"569c42c8-855a-4208-982a-b238ddaf3ee9\" (UID: \"569c42c8-855a-4208-982a-b238ddaf3ee9\") " Aug 03 23:37:58.431662 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431566 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-service-ca" (OuterVolumeSpecName: "service-ca") pod "569c42c8-855a-4208-982a-b238ddaf3ee9" (UID: "569c42c8-855a-4208-982a-b238ddaf3ee9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:37:58.431721 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431700 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-console-config" (OuterVolumeSpecName: "console-config") pod "569c42c8-855a-4208-982a-b238ddaf3ee9" (UID: "569c42c8-855a-4208-982a-b238ddaf3ee9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:37:58.431774 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431725 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:37:58.431843 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431813 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "569c42c8-855a-4208-982a-b238ddaf3ee9" (UID: "569c42c8-855a-4208-982a-b238ddaf3ee9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:37:58.431990 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.431958 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "569c42c8-855a-4208-982a-b238ddaf3ee9" (UID: "569c42c8-855a-4208-982a-b238ddaf3ee9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:37:58.433483 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.433434 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "569c42c8-855a-4208-982a-b238ddaf3ee9" (UID: "569c42c8-855a-4208-982a-b238ddaf3ee9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 03 23:37:58.433623 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.433606 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569c42c8-855a-4208-982a-b238ddaf3ee9-kube-api-access-ljbl4" (OuterVolumeSpecName: "kube-api-access-ljbl4") pod "569c42c8-855a-4208-982a-b238ddaf3ee9" (UID: "569c42c8-855a-4208-982a-b238ddaf3ee9"). InnerVolumeSpecName "kube-api-access-ljbl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:37:58.433671 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.433660 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "569c42c8-855a-4208-982a-b238ddaf3ee9" (UID: "569c42c8-855a-4208-982a-b238ddaf3ee9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 03 23:37:58.532416 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.532360 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-oauth-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:37:58.532416 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.532411 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:37:58.532416 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.532425 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-console-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:37:58.532705 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.532440 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569c42c8-855a-4208-982a-b238ddaf3ee9-trusted-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:37:58.532705 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.532464 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/569c42c8-855a-4208-982a-b238ddaf3ee9-console-oauth-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:37:58.532705 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:58.532478 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljbl4\" (UniqueName: \"kubernetes.io/projected/569c42c8-855a-4208-982a-b238ddaf3ee9-kube-api-access-ljbl4\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:37:59.139817 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.139777 2556 generic.go:356] "Generic (PLEG): container finished" podID="569c42c8-855a-4208-982a-b238ddaf3ee9" containerID="f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f" exitCode=0 Aug 03 23:37:59.140288 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.139831 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fffdb9589-xx4db" event={"ID":"569c42c8-855a-4208-982a-b238ddaf3ee9","Type":"ContainerDied","Data":"f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f"} Aug 03 23:37:59.140288 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.139856 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fffdb9589-xx4db" Aug 03 23:37:59.140288 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.139947 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fffdb9589-xx4db" event={"ID":"569c42c8-855a-4208-982a-b238ddaf3ee9","Type":"ContainerDied","Data":"9b59b5db5347f5bea85dab3527cc0d2d707733580a4eb5e702d6166a1c20dd29"} Aug 03 23:37:59.140288 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.139982 2556 scope.go:117] "RemoveContainer" containerID="f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f" Aug 03 23:37:59.149026 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.149008 2556 scope.go:117] "RemoveContainer" containerID="f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f" Aug 03 23:37:59.149348 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:37:59.149326 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f\": container with ID starting with f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f not found: ID does not exist" containerID="f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f" Aug 03 23:37:59.149413 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.149357 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f"} err="failed to get container status \"f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f\": rpc error: code = NotFound desc = could not find container \"f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f\": container with ID starting with f12c3e4fb7297f5252789215c88f2f7867f839b964069431b66c0a5e5adecd6f not found: ID does not exist" Aug 03 23:37:59.160783 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.160749 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fffdb9589-xx4db"] Aug 03 23:37:59.164569 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:37:59.164544 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fffdb9589-xx4db"] Aug 03 23:38:00.420149 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:00.420118 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569c42c8-855a-4208-982a-b238ddaf3ee9" path="/var/lib/kubelet/pods/569c42c8-855a-4208-982a-b238ddaf3ee9/volumes" Aug 03 23:38:23.137683 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.137650 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s"] Aug 03 23:38:23.138060 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.137983 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569c42c8-855a-4208-982a-b238ddaf3ee9" containerName="console" Aug 03 23:38:23.138060 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.137995 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="569c42c8-855a-4208-982a-b238ddaf3ee9" containerName="console" Aug 03 23:38:23.138130 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.138073 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="569c42c8-855a-4208-982a-b238ddaf3ee9" containerName="console" Aug 03 23:38:23.141168 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.141148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.143692 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.143671 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fr74m\"" Aug 03 23:38:23.143792 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.143671 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 03 23:38:23.143792 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.143713 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 03 23:38:23.149102 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.149077 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s"] Aug 03 23:38:23.207992 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.207962 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.208132 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.207997 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.208132 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.208021 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbnm\" (UniqueName: \"kubernetes.io/projected/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-kube-api-access-lcbnm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.309250 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.309215 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.309250 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.309249 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.309475 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.309272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbnm\" (UniqueName: \"kubernetes.io/projected/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-kube-api-access-lcbnm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.309621 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.309602 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.309676 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.309637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.318131 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.318100 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbnm\" (UniqueName: \"kubernetes.io/projected/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-kube-api-access-lcbnm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.451242 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.451169 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:23.584025 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:23.584001 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s"] Aug 03 23:38:23.586564 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:38:23.586532 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d7d8f9_b167_46fa_a29f_d3fec210c0c9.slice/crio-122a2d5f05ef9c6961bb1af289df9369b4328ba8c3f01e133b9d984be880bcfb WatchSource:0}: Error finding container 122a2d5f05ef9c6961bb1af289df9369b4328ba8c3f01e133b9d984be880bcfb: Status 404 returned error can't find the container with id 122a2d5f05ef9c6961bb1af289df9369b4328ba8c3f01e133b9d984be880bcfb Aug 03 23:38:24.245520 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:24.245478 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" event={"ID":"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9","Type":"ContainerStarted","Data":"122a2d5f05ef9c6961bb1af289df9369b4328ba8c3f01e133b9d984be880bcfb"} Aug 03 23:38:30.273522 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:30.273481 2556 generic.go:356] "Generic (PLEG): container finished" podID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerID="e584069937df66ae50db60fea046392833b9ba21e2bd62546e048f93df01c497" exitCode=0 Aug 03 23:38:30.274055 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:30.273582 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" event={"ID":"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9","Type":"ContainerDied","Data":"e584069937df66ae50db60fea046392833b9ba21e2bd62546e048f93df01c497"} Aug 03 23:38:37.303329 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:37.303267 2556 generic.go:356] "Generic (PLEG): container finished" podID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerID="add1c5ee87593a8933443f17653bfc30d47e8377639edff263770a816241a30b" exitCode=0 Aug 03 23:38:37.303819 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:37.303360 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" event={"ID":"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9","Type":"ContainerDied","Data":"add1c5ee87593a8933443f17653bfc30d47e8377639edff263770a816241a30b"} Aug 03 23:38:44.337850 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:44.337815 2556 generic.go:356] "Generic (PLEG): container finished" podID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerID="fb38f3274de91da70c157a4e7bd94006f79f4475c6fea556c449564c66bade9a" exitCode=0 Aug 03 23:38:44.338334 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:44.337910 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" event={"ID":"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9","Type":"ContainerDied","Data":"fb38f3274de91da70c157a4e7bd94006f79f4475c6fea556c449564c66bade9a"} Aug 03 23:38:45.518393 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.518365 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:45.579832 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.579808 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcbnm\" (UniqueName: \"kubernetes.io/projected/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-kube-api-access-lcbnm\") pod \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " Aug 03 23:38:45.579975 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.579849 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-bundle\") pod \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " Aug 03 23:38:45.579975 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.579912 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-util\") pod \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\" (UID: \"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9\") " Aug 03 23:38:45.580583 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.580553 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-bundle" (OuterVolumeSpecName: "bundle") pod "c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" (UID: "c8d7d8f9-b167-46fa-a29f-d3fec210c0c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:38:45.582099 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.582068 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-kube-api-access-lcbnm" (OuterVolumeSpecName: "kube-api-access-lcbnm") pod "c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" (UID: "c8d7d8f9-b167-46fa-a29f-d3fec210c0c9"). InnerVolumeSpecName "kube-api-access-lcbnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:38:45.584994 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.584973 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-util" (OuterVolumeSpecName: "util") pod "c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" (UID: "c8d7d8f9-b167-46fa-a29f-d3fec210c0c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:38:45.680854 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.680788 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-util\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:38:45.680854 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.680815 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcbnm\" (UniqueName: \"kubernetes.io/projected/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-kube-api-access-lcbnm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:38:45.680854 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:45.680826 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8d7d8f9-b167-46fa-a29f-d3fec210c0c9-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:38:46.348713 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:46.348687 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" Aug 03 23:38:46.348879 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:46.348686 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpf9s" event={"ID":"c8d7d8f9-b167-46fa-a29f-d3fec210c0c9","Type":"ContainerDied","Data":"122a2d5f05ef9c6961bb1af289df9369b4328ba8c3f01e133b9d984be880bcfb"} Aug 03 23:38:46.348879 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:46.348837 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="122a2d5f05ef9c6961bb1af289df9369b4328ba8c3f01e133b9d984be880bcfb" Aug 03 23:38:51.009781 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.009745 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7"] Aug 03 23:38:51.010272 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.010040 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerName="pull" Aug 03 23:38:51.010272 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.010050 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerName="pull" Aug 03 23:38:51.010272 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.010064 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerName="extract" Aug 03 23:38:51.010272 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.010071 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerName="extract" Aug 03 23:38:51.010272 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.010088 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerName="util" Aug 03 23:38:51.010272 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.010093 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerName="util" Aug 03 23:38:51.010272 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.010139 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d7d8f9-b167-46fa-a29f-d3fec210c0c9" containerName="extract" Aug 03 23:38:51.012803 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.012787 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" Aug 03 23:38:51.015343 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.015298 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-48888\"" Aug 03 23:38:51.015343 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.015298 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 03 23:38:51.015511 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.015298 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 03 23:38:51.035548 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.035527 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7"] Aug 03 23:38:51.120198 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.120167 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rttl\" (UniqueName: \"kubernetes.io/projected/1c376247-65aa-4ff8-8231-2f3a630f518f-kube-api-access-2rttl\") pod \"cert-manager-operator-controller-manager-55c444bf86-bz6n7\" (UID: \"1c376247-65aa-4ff8-8231-2f3a630f518f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" Aug 03 23:38:51.120360 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.120209 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c376247-65aa-4ff8-8231-2f3a630f518f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bz6n7\" (UID: \"1c376247-65aa-4ff8-8231-2f3a630f518f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" Aug 03 23:38:51.221210 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.221182 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rttl\" (UniqueName: \"kubernetes.io/projected/1c376247-65aa-4ff8-8231-2f3a630f518f-kube-api-access-2rttl\") pod \"cert-manager-operator-controller-manager-55c444bf86-bz6n7\" (UID: \"1c376247-65aa-4ff8-8231-2f3a630f518f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" Aug 03 23:38:51.221388 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.221216 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c376247-65aa-4ff8-8231-2f3a630f518f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bz6n7\" (UID: \"1c376247-65aa-4ff8-8231-2f3a630f518f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" Aug 03 23:38:51.221584 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.221568 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c376247-65aa-4ff8-8231-2f3a630f518f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bz6n7\" (UID: \"1c376247-65aa-4ff8-8231-2f3a630f518f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" Aug 03 23:38:51.230087 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.230068 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rttl\" (UniqueName: \"kubernetes.io/projected/1c376247-65aa-4ff8-8231-2f3a630f518f-kube-api-access-2rttl\") pod \"cert-manager-operator-controller-manager-55c444bf86-bz6n7\" (UID: \"1c376247-65aa-4ff8-8231-2f3a630f518f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" Aug 03 23:38:51.322737 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.322666 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" Aug 03 23:38:51.461871 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:51.461849 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7"] Aug 03 23:38:51.464640 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:38:51.464615 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c376247_65aa_4ff8_8231_2f3a630f518f.slice/crio-905fe0556c54a59d9d7dbf08cba7a73288cd57587fb076223922aff67aae2f1c WatchSource:0}: Error finding container 905fe0556c54a59d9d7dbf08cba7a73288cd57587fb076223922aff67aae2f1c: Status 404 returned error can't find the container with id 905fe0556c54a59d9d7dbf08cba7a73288cd57587fb076223922aff67aae2f1c Aug 03 23:38:52.377010 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:52.376973 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" event={"ID":"1c376247-65aa-4ff8-8231-2f3a630f518f","Type":"ContainerStarted","Data":"905fe0556c54a59d9d7dbf08cba7a73288cd57587fb076223922aff67aae2f1c"} Aug 03 23:38:53.382628 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:53.382589 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" event={"ID":"1c376247-65aa-4ff8-8231-2f3a630f518f","Type":"ContainerStarted","Data":"4497807dfe5e70581dabdbaf1f939f794032f024de43c77acc00bbbba2b6d9ee"} Aug 03 23:38:53.403158 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:53.403061 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bz6n7" podStartSLOduration=1.723466142 podStartE2EDuration="3.403047295s" totalImagesPullingTime="1.679581153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:38:53.40164263 +0000 UTC m=+547.464899060" watchObservedRunningTime="2026-08-03 23:38:53.403047295 +0000 UTC m=+547.466303724" Aug 03 23:38:54.739099 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.739063 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m"] Aug 03 23:38:54.741514 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.741496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.744094 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.744068 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 03 23:38:54.744226 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.744068 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 03 23:38:54.745122 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.745100 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fr74m\"" Aug 03 23:38:54.752992 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.752966 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m"] Aug 03 23:38:54.838850 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.838819 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qr8k7"] Aug 03 23:38:54.840970 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.840953 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:54.843235 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.843196 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 03 23:38:54.843235 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.843231 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 03 23:38:54.843456 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.843275 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7288n\"" Aug 03 23:38:54.849027 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.849005 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qr8k7"] Aug 03 23:38:54.854141 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.854116 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.854256 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.854166 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.854256 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.854204 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vbp\" (UniqueName: \"kubernetes.io/projected/106230cd-29e8-4396-8907-74c598cbc1e7-kube-api-access-24vbp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.955278 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.955243 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.955278 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.955282 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vbp\" (UniqueName: \"kubernetes.io/projected/106230cd-29e8-4396-8907-74c598cbc1e7-kube-api-access-24vbp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.955509 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.955322 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4l6m\" (UniqueName: \"kubernetes.io/projected/73f3a2a5-40e7-465d-9844-b7ea35e31dba-kube-api-access-b4l6m\") pod \"cert-manager-webhook-6486ffd997-qr8k7\" (UID: \"73f3a2a5-40e7-465d-9844-b7ea35e31dba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:54.955509 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.955357 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.955509 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.955382 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73f3a2a5-40e7-465d-9844-b7ea35e31dba-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qr8k7\" (UID: \"73f3a2a5-40e7-465d-9844-b7ea35e31dba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:54.955670 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.955650 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.955710 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.955674 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:54.963965 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:54.963937 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vbp\" (UniqueName: \"kubernetes.io/projected/106230cd-29e8-4396-8907-74c598cbc1e7-kube-api-access-24vbp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:55.053655 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.053584 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:38:55.055942 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.055662 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4l6m\" (UniqueName: \"kubernetes.io/projected/73f3a2a5-40e7-465d-9844-b7ea35e31dba-kube-api-access-b4l6m\") pod \"cert-manager-webhook-6486ffd997-qr8k7\" (UID: \"73f3a2a5-40e7-465d-9844-b7ea35e31dba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:55.055942 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.055710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73f3a2a5-40e7-465d-9844-b7ea35e31dba-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qr8k7\" (UID: \"73f3a2a5-40e7-465d-9844-b7ea35e31dba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:55.065486 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.065448 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73f3a2a5-40e7-465d-9844-b7ea35e31dba-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qr8k7\" (UID: \"73f3a2a5-40e7-465d-9844-b7ea35e31dba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:55.065636 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.065614 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4l6m\" (UniqueName: \"kubernetes.io/projected/73f3a2a5-40e7-465d-9844-b7ea35e31dba-kube-api-access-b4l6m\") pod \"cert-manager-webhook-6486ffd997-qr8k7\" (UID: \"73f3a2a5-40e7-465d-9844-b7ea35e31dba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:55.159747 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.159710 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:55.185042 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.185021 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m"] Aug 03 23:38:55.187946 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:38:55.187911 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106230cd_29e8_4396_8907_74c598cbc1e7.slice/crio-90107559b11ce2bc78313631a132c833180b2654034756de9c12ef79a4b54fc7 WatchSource:0}: Error finding container 90107559b11ce2bc78313631a132c833180b2654034756de9c12ef79a4b54fc7: Status 404 returned error can't find the container with id 90107559b11ce2bc78313631a132c833180b2654034756de9c12ef79a4b54fc7 Aug 03 23:38:55.301998 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.301971 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qr8k7"] Aug 03 23:38:55.346208 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:38:55.346151 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f3a2a5_40e7_465d_9844_b7ea35e31dba.slice/crio-65fe218040467305021d8d20615eeff4d8b10d99c0b37c3c6cc7ed483a308e6d WatchSource:0}: Error finding container 65fe218040467305021d8d20615eeff4d8b10d99c0b37c3c6cc7ed483a308e6d: Status 404 returned error can't find the container with id 65fe218040467305021d8d20615eeff4d8b10d99c0b37c3c6cc7ed483a308e6d Aug 03 23:38:55.392342 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.392294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" event={"ID":"73f3a2a5-40e7-465d-9844-b7ea35e31dba","Type":"ContainerStarted","Data":"65fe218040467305021d8d20615eeff4d8b10d99c0b37c3c6cc7ed483a308e6d"} Aug 03 23:38:55.393817 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.393794 2556 generic.go:356] "Generic (PLEG): container finished" podID="106230cd-29e8-4396-8907-74c598cbc1e7" containerID="e4cf4175bfaa6ab762969461f4ad93e52d3b7287c864bc98972ebba7525849d4" exitCode=0 Aug 03 23:38:55.393908 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.393828 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" event={"ID":"106230cd-29e8-4396-8907-74c598cbc1e7","Type":"ContainerDied","Data":"e4cf4175bfaa6ab762969461f4ad93e52d3b7287c864bc98972ebba7525849d4"} Aug 03 23:38:55.393908 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:55.393858 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" event={"ID":"106230cd-29e8-4396-8907-74c598cbc1e7","Type":"ContainerStarted","Data":"90107559b11ce2bc78313631a132c833180b2654034756de9c12ef79a4b54fc7"} Aug 03 23:38:57.192557 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.192482 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7"] Aug 03 23:38:57.200247 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.199959 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" Aug 03 23:38:57.202904 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.202869 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-czgw9\"" Aug 03 23:38:57.203586 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.203556 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7"] Aug 03 23:38:57.273393 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.273356 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2tt\" (UniqueName: \"kubernetes.io/projected/4341e77a-390a-4769-8aca-8a550501559b-kube-api-access-7h2tt\") pod \"cert-manager-cainjector-559cdd8c87-bgrb7\" (UID: \"4341e77a-390a-4769-8aca-8a550501559b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" Aug 03 23:38:57.273580 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.273543 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4341e77a-390a-4769-8aca-8a550501559b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgrb7\" (UID: \"4341e77a-390a-4769-8aca-8a550501559b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" Aug 03 23:38:57.374499 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.374457 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4341e77a-390a-4769-8aca-8a550501559b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgrb7\" (UID: \"4341e77a-390a-4769-8aca-8a550501559b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" Aug 03 23:38:57.374665 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.374537 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2tt\" (UniqueName: \"kubernetes.io/projected/4341e77a-390a-4769-8aca-8a550501559b-kube-api-access-7h2tt\") pod \"cert-manager-cainjector-559cdd8c87-bgrb7\" (UID: \"4341e77a-390a-4769-8aca-8a550501559b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" Aug 03 23:38:57.386461 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.386419 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4341e77a-390a-4769-8aca-8a550501559b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bgrb7\" (UID: \"4341e77a-390a-4769-8aca-8a550501559b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" Aug 03 23:38:57.386645 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.386621 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2tt\" (UniqueName: \"kubernetes.io/projected/4341e77a-390a-4769-8aca-8a550501559b-kube-api-access-7h2tt\") pod \"cert-manager-cainjector-559cdd8c87-bgrb7\" (UID: \"4341e77a-390a-4769-8aca-8a550501559b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" Aug 03 23:38:57.512695 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.512666 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" Aug 03 23:38:57.662281 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:57.662141 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7"] Aug 03 23:38:58.411847 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:58.411810 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" event={"ID":"106230cd-29e8-4396-8907-74c598cbc1e7","Type":"ContainerStarted","Data":"d5ba653f75ad992a56691440d9c95123fc4ebe04e5d08b4fe28c3857b57ed6bf"} Aug 03 23:38:58.569601 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:38:58.569563 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4341e77a_390a_4769_8aca_8a550501559b.slice/crio-cca82ea1c8245e4facbb1e1a0fc0b5afb8a72ea35c6a2a21007961cb92cf77da WatchSource:0}: Error finding container cca82ea1c8245e4facbb1e1a0fc0b5afb8a72ea35c6a2a21007961cb92cf77da: Status 404 returned error can't find the container with id cca82ea1c8245e4facbb1e1a0fc0b5afb8a72ea35c6a2a21007961cb92cf77da Aug 03 23:38:59.418607 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:59.418575 2556 generic.go:356] "Generic (PLEG): container finished" podID="106230cd-29e8-4396-8907-74c598cbc1e7" containerID="d5ba653f75ad992a56691440d9c95123fc4ebe04e5d08b4fe28c3857b57ed6bf" exitCode=0 Aug 03 23:38:59.419004 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:59.418662 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" event={"ID":"106230cd-29e8-4396-8907-74c598cbc1e7","Type":"ContainerDied","Data":"d5ba653f75ad992a56691440d9c95123fc4ebe04e5d08b4fe28c3857b57ed6bf"} Aug 03 23:38:59.420393 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:59.420334 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" event={"ID":"73f3a2a5-40e7-465d-9844-b7ea35e31dba","Type":"ContainerStarted","Data":"d1be8d0857737525feea19611d949f66479053b6f3e3fdb16bd831befad2e790"} Aug 03 23:38:59.420469 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:59.420437 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:38:59.421959 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:59.421934 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" event={"ID":"4341e77a-390a-4769-8aca-8a550501559b","Type":"ContainerStarted","Data":"07a28ba5afd3a43251b56e04399585a1461e77e47923fdb68c5484eefeb77c77"} Aug 03 23:38:59.422041 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:59.421966 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" event={"ID":"4341e77a-390a-4769-8aca-8a550501559b","Type":"ContainerStarted","Data":"cca82ea1c8245e4facbb1e1a0fc0b5afb8a72ea35c6a2a21007961cb92cf77da"} Aug 03 23:38:59.449836 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:59.449765 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bgrb7" podStartSLOduration=1.832760918 podStartE2EDuration="2.449749907s" totalImagesPullingTime="616.988989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:38:59.448894961 +0000 UTC m=+553.512151388" watchObservedRunningTime="2026-08-03 23:38:59.449749907 +0000 UTC m=+553.513006336" Aug 03 23:38:59.463127 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:38:59.463075 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" podStartSLOduration=2.208770919 podStartE2EDuration="5.463056862s" totalImagesPullingTime="3.254285943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:38:59.462447524 +0000 UTC m=+553.525703955" watchObservedRunningTime="2026-08-03 23:38:59.463056862 +0000 UTC m=+553.526313293" Aug 03 23:39:00.427389 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:00.427354 2556 generic.go:356] "Generic (PLEG): container finished" podID="106230cd-29e8-4396-8907-74c598cbc1e7" containerID="dd9c5e7cc01492b48dd2bf965570a7aaba770df56d851e91045bbbe71fbd7628" exitCode=0 Aug 03 23:39:00.427809 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:00.427439 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" event={"ID":"106230cd-29e8-4396-8907-74c598cbc1e7","Type":"ContainerDied","Data":"dd9c5e7cc01492b48dd2bf965570a7aaba770df56d851e91045bbbe71fbd7628"} Aug 03 23:39:01.564369 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.564335 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:39:01.712476 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.712448 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-util\") pod \"106230cd-29e8-4396-8907-74c598cbc1e7\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " Aug 03 23:39:01.712648 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.712515 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24vbp\" (UniqueName: \"kubernetes.io/projected/106230cd-29e8-4396-8907-74c598cbc1e7-kube-api-access-24vbp\") pod \"106230cd-29e8-4396-8907-74c598cbc1e7\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " Aug 03 23:39:01.712648 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.712578 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-bundle\") pod \"106230cd-29e8-4396-8907-74c598cbc1e7\" (UID: \"106230cd-29e8-4396-8907-74c598cbc1e7\") " Aug 03 23:39:01.712932 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.712905 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-bundle" (OuterVolumeSpecName: "bundle") pod "106230cd-29e8-4396-8907-74c598cbc1e7" (UID: "106230cd-29e8-4396-8907-74c598cbc1e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:39:01.714633 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.714609 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106230cd-29e8-4396-8907-74c598cbc1e7-kube-api-access-24vbp" (OuterVolumeSpecName: "kube-api-access-24vbp") pod "106230cd-29e8-4396-8907-74c598cbc1e7" (UID: "106230cd-29e8-4396-8907-74c598cbc1e7"). InnerVolumeSpecName "kube-api-access-24vbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:39:01.716540 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.716519 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-util" (OuterVolumeSpecName: "util") pod "106230cd-29e8-4396-8907-74c598cbc1e7" (UID: "106230cd-29e8-4396-8907-74c598cbc1e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:39:01.813970 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.813943 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-util\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:39:01.813970 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.813967 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vbp\" (UniqueName: \"kubernetes.io/projected/106230cd-29e8-4396-8907-74c598cbc1e7-kube-api-access-24vbp\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:39:01.814127 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:01.813977 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106230cd-29e8-4396-8907-74c598cbc1e7-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:39:02.439239 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:02.439210 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" Aug 03 23:39:02.439394 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:02.439215 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr7c4m" event={"ID":"106230cd-29e8-4396-8907-74c598cbc1e7","Type":"ContainerDied","Data":"90107559b11ce2bc78313631a132c833180b2654034756de9c12ef79a4b54fc7"} Aug 03 23:39:02.439394 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:02.439346 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90107559b11ce2bc78313631a132c833180b2654034756de9c12ef79a4b54fc7" Aug 03 23:39:05.429643 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:05.429615 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qr8k7" Aug 03 23:39:15.554575 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.554537 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm"] Aug 03 23:39:15.555065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.554843 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106230cd-29e8-4396-8907-74c598cbc1e7" containerName="pull" Aug 03 23:39:15.555065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.554854 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="106230cd-29e8-4396-8907-74c598cbc1e7" containerName="pull" Aug 03 23:39:15.555065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.554864 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106230cd-29e8-4396-8907-74c598cbc1e7" containerName="extract" Aug 03 23:39:15.555065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.554870 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="106230cd-29e8-4396-8907-74c598cbc1e7" containerName="extract" Aug 03 23:39:15.555065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.554885 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106230cd-29e8-4396-8907-74c598cbc1e7" containerName="util" Aug 03 23:39:15.555065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.554890 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="106230cd-29e8-4396-8907-74c598cbc1e7" containerName="util" Aug 03 23:39:15.555065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.554948 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="106230cd-29e8-4396-8907-74c598cbc1e7" containerName="extract" Aug 03 23:39:15.558999 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.558980 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.561555 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.561533 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 03 23:39:15.561679 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.561632 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fr74m\"" Aug 03 23:39:15.562644 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.562625 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 03 23:39:15.566171 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.566148 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm"] Aug 03 23:39:15.738105 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.738072 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.738283 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.738122 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.738393 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.738277 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdxk\" (UniqueName: \"kubernetes.io/projected/365674a9-7eb3-44d9-85cf-5718e2bf5787-kube-api-access-lwdxk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.839596 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.839507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.839596 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.839585 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdxk\" (UniqueName: \"kubernetes.io/projected/365674a9-7eb3-44d9-85cf-5718e2bf5787-kube-api-access-lwdxk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.839799 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.839613 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.839945 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.839923 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.840007 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.839933 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.847927 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.847903 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdxk\" (UniqueName: \"kubernetes.io/projected/365674a9-7eb3-44d9-85cf-5718e2bf5787-kube-api-access-lwdxk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.869594 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.869574 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:15.996566 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:15.996539 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm"] Aug 03 23:39:16.503699 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:16.503668 2556 generic.go:356] "Generic (PLEG): container finished" podID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerID="60c4041bd376ae72f6dbebd1f412f8848cf06552cb350375f532451a0c853267" exitCode=0 Aug 03 23:39:16.503891 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:16.503744 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" event={"ID":"365674a9-7eb3-44d9-85cf-5718e2bf5787","Type":"ContainerDied","Data":"60c4041bd376ae72f6dbebd1f412f8848cf06552cb350375f532451a0c853267"} Aug 03 23:39:16.503891 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:16.503778 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" event={"ID":"365674a9-7eb3-44d9-85cf-5718e2bf5787","Type":"ContainerStarted","Data":"be3bf8aae81377b35977a0bb48ae2cff8ea7eb36d707e0a5d213ed8481f5fdb7"} Aug 03 23:39:17.509826 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:17.509794 2556 generic.go:356] "Generic (PLEG): container finished" podID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerID="e90f2c9140d468f7756fab8e6f06895bc262e9d492afb33d37b46f08722e8573" exitCode=0 Aug 03 23:39:17.510261 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:17.509877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" event={"ID":"365674a9-7eb3-44d9-85cf-5718e2bf5787","Type":"ContainerDied","Data":"e90f2c9140d468f7756fab8e6f06895bc262e9d492afb33d37b46f08722e8573"} Aug 03 23:39:18.519066 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:18.519032 2556 generic.go:356] "Generic (PLEG): container finished" podID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerID="347dcfa18f54eb2c0887a65292df1e438ef824e1d7f8107c5717082b9126636b" exitCode=0 Aug 03 23:39:18.519488 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:18.519107 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" event={"ID":"365674a9-7eb3-44d9-85cf-5718e2bf5787","Type":"ContainerDied","Data":"347dcfa18f54eb2c0887a65292df1e438ef824e1d7f8107c5717082b9126636b"} Aug 03 23:39:19.650230 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.650207 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:19.772913 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.772889 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-util\") pod \"365674a9-7eb3-44d9-85cf-5718e2bf5787\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " Aug 03 23:39:19.773065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.772956 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwdxk\" (UniqueName: \"kubernetes.io/projected/365674a9-7eb3-44d9-85cf-5718e2bf5787-kube-api-access-lwdxk\") pod \"365674a9-7eb3-44d9-85cf-5718e2bf5787\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " Aug 03 23:39:19.773065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.772987 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-bundle\") pod \"365674a9-7eb3-44d9-85cf-5718e2bf5787\" (UID: \"365674a9-7eb3-44d9-85cf-5718e2bf5787\") " Aug 03 23:39:19.773777 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.773744 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-bundle" (OuterVolumeSpecName: "bundle") pod "365674a9-7eb3-44d9-85cf-5718e2bf5787" (UID: "365674a9-7eb3-44d9-85cf-5718e2bf5787"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:39:19.775114 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.775055 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365674a9-7eb3-44d9-85cf-5718e2bf5787-kube-api-access-lwdxk" (OuterVolumeSpecName: "kube-api-access-lwdxk") pod "365674a9-7eb3-44d9-85cf-5718e2bf5787" (UID: "365674a9-7eb3-44d9-85cf-5718e2bf5787"). InnerVolumeSpecName "kube-api-access-lwdxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:39:19.781216 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.781188 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-util" (OuterVolumeSpecName: "util") pod "365674a9-7eb3-44d9-85cf-5718e2bf5787" (UID: "365674a9-7eb3-44d9-85cf-5718e2bf5787"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:39:19.873548 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.873515 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwdxk\" (UniqueName: \"kubernetes.io/projected/365674a9-7eb3-44d9-85cf-5718e2bf5787-kube-api-access-lwdxk\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:39:19.873548 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.873543 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:39:19.873548 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:19.873551 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/365674a9-7eb3-44d9-85cf-5718e2bf5787-util\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:39:20.530283 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:20.530254 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" Aug 03 23:39:20.530461 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:20.530255 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjt7nm" event={"ID":"365674a9-7eb3-44d9-85cf-5718e2bf5787","Type":"ContainerDied","Data":"be3bf8aae81377b35977a0bb48ae2cff8ea7eb36d707e0a5d213ed8481f5fdb7"} Aug 03 23:39:20.530461 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:20.530361 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be3bf8aae81377b35977a0bb48ae2cff8ea7eb36d707e0a5d213ed8481f5fdb7" Aug 03 23:39:28.667606 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.667572 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v"] Aug 03 23:39:28.667979 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.667898 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerName="extract" Aug 03 23:39:28.667979 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.667910 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerName="extract" Aug 03 23:39:28.667979 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.667918 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerName="pull" Aug 03 23:39:28.667979 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.667924 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerName="pull" Aug 03 23:39:28.667979 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.667933 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerName="util" Aug 03 23:39:28.667979 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.667938 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerName="util" Aug 03 23:39:28.668160 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.667991 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="365674a9-7eb3-44d9-85cf-5718e2bf5787" containerName="extract" Aug 03 23:39:28.670158 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.670143 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.672763 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.672725 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 03 23:39:28.672919 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.672762 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 03 23:39:28.672919 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.672809 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 03 23:39:28.672919 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.672815 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 03 23:39:28.672919 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.672810 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hcprk\"" Aug 03 23:39:28.686065 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.686041 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v"] Aug 03 23:39:28.734316 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.734274 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqk6c\" (UniqueName: \"kubernetes.io/projected/a8a09e21-d143-472d-bfc4-58719aef68f2-kube-api-access-gqk6c\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.734441 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.734332 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8a09e21-d143-472d-bfc4-58719aef68f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.734441 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.734389 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a09e21-d143-472d-bfc4-58719aef68f2-tmp\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.734513 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.734444 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8a09e21-d143-472d-bfc4-58719aef68f2-manifests\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.734513 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.734469 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8a09e21-d143-472d-bfc4-58719aef68f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.835263 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.835233 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk6c\" (UniqueName: \"kubernetes.io/projected/a8a09e21-d143-472d-bfc4-58719aef68f2-kube-api-access-gqk6c\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.835428 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.835269 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8a09e21-d143-472d-bfc4-58719aef68f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.835428 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.835329 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a09e21-d143-472d-bfc4-58719aef68f2-tmp\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.835428 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.835384 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8a09e21-d143-472d-bfc4-58719aef68f2-manifests\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.835428 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.835417 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8a09e21-d143-472d-bfc4-58719aef68f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.835849 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.835817 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8a09e21-d143-472d-bfc4-58719aef68f2-manifests\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.837916 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.837885 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a09e21-d143-472d-bfc4-58719aef68f2-tmp\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.838005 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.837940 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8a09e21-d143-472d-bfc4-58719aef68f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.838056 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.838038 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8a09e21-d143-472d-bfc4-58719aef68f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.844137 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.844111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk6c\" (UniqueName: \"kubernetes.io/projected/a8a09e21-d143-472d-bfc4-58719aef68f2-kube-api-access-gqk6c\") pod \"opendatahub-operator-controller-manager-6869c6d9bb-mk77v\" (UID: \"a8a09e21-d143-472d-bfc4-58719aef68f2\") " pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:28.980790 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:28.980760 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:29.116415 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:29.116388 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v"] Aug 03 23:39:29.118408 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:39:29.118380 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a09e21_d143_472d_bfc4_58719aef68f2.slice/crio-bf13f68ab2dd8c7a0d916a0f5c3726a02430a45ef4beeab09be1c16a151c7a72 WatchSource:0}: Error finding container bf13f68ab2dd8c7a0d916a0f5c3726a02430a45ef4beeab09be1c16a151c7a72: Status 404 returned error can't find the container with id bf13f68ab2dd8c7a0d916a0f5c3726a02430a45ef4beeab09be1c16a151c7a72 Aug 03 23:39:29.571547 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:29.571508 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" event={"ID":"a8a09e21-d143-472d-bfc4-58719aef68f2","Type":"ContainerStarted","Data":"bf13f68ab2dd8c7a0d916a0f5c3726a02430a45ef4beeab09be1c16a151c7a72"} Aug 03 23:39:32.588950 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:32.588915 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" event={"ID":"a8a09e21-d143-472d-bfc4-58719aef68f2","Type":"ContainerStarted","Data":"80e7552b530c02900665c1ff5a1693ffba2edad68d588de36473214109e27b33"} Aug 03 23:39:33.594974 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:33.594885 2556 generic.go:356] "Generic (PLEG): container finished" podID="a8a09e21-d143-472d-bfc4-58719aef68f2" containerID="80e7552b530c02900665c1ff5a1693ffba2edad68d588de36473214109e27b33" exitCode=0 Aug 03 23:39:33.595402 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:33.594975 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" event={"ID":"a8a09e21-d143-472d-bfc4-58719aef68f2","Type":"ContainerDied","Data":"80e7552b530c02900665c1ff5a1693ffba2edad68d588de36473214109e27b33"} Aug 03 23:39:34.602385 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:34.602294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" event={"ID":"a8a09e21-d143-472d-bfc4-58719aef68f2","Type":"ContainerStarted","Data":"2902c95609282e1c89dbaacd08bf7d60ea1f7aa58103b27a8b55d80ccca0cd6a"} Aug 03 23:39:34.602809 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:34.602495 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:34.624700 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:34.624653 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" podStartSLOduration=2.18159989 podStartE2EDuration="6.624639439s" totalImagesPullingTime="3.36263057s" totalInitContainerRuntime="1.080408979s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:39:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:39:34.621909393 +0000 UTC m=+588.685165825" watchObservedRunningTime="2026-08-03 23:39:34.624639439 +0000 UTC m=+588.687895868" Aug 03 23:39:45.609727 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.609696 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6869c6d9bb-mk77v" Aug 03 23:39:45.826784 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.826746 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bdbb769bb-9vfpm"] Aug 03 23:39:45.834098 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.834072 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.843071 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.843048 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdbb769bb-9vfpm"] Aug 03 23:39:45.878627 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.878525 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-serving-cert\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.878627 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.878577 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-oauth-config\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.878838 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.878631 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-service-ca\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.878838 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.878658 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-oauth-serving-cert\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.878838 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.878765 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-trusted-ca-bundle\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.878838 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.878818 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-config\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.879048 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.878844 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn4jc\" (UniqueName: \"kubernetes.io/projected/031ccbfe-ab72-4869-82f1-076f05d4dcf4-kube-api-access-pn4jc\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.979865 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.979827 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-trusted-ca-bundle\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.979865 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.979870 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-config\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.980074 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.979888 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn4jc\" (UniqueName: \"kubernetes.io/projected/031ccbfe-ab72-4869-82f1-076f05d4dcf4-kube-api-access-pn4jc\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.980112 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.980082 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-serving-cert\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.980157 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.980126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-oauth-config\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.980213 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.980164 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-service-ca\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.980213 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.980190 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-oauth-serving-cert\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.980802 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.980776 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-config\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.980995 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.980973 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-oauth-serving-cert\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.981048 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.980974 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-service-ca\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.981098 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.981019 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031ccbfe-ab72-4869-82f1-076f05d4dcf4-trusted-ca-bundle\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.982714 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.982693 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-oauth-config\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.982819 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.982800 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/031ccbfe-ab72-4869-82f1-076f05d4dcf4-console-serving-cert\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:45.988542 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:45.988518 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn4jc\" (UniqueName: \"kubernetes.io/projected/031ccbfe-ab72-4869-82f1-076f05d4dcf4-kube-api-access-pn4jc\") pod \"console-bdbb769bb-9vfpm\" (UID: \"031ccbfe-ab72-4869-82f1-076f05d4dcf4\") " pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:46.145549 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:46.145460 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:46.283553 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:46.282480 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdbb769bb-9vfpm"] Aug 03 23:39:46.560915 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:46.560797 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:39:46.560915 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:46.560853 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:39:46.658292 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:46.658260 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdbb769bb-9vfpm" event={"ID":"031ccbfe-ab72-4869-82f1-076f05d4dcf4","Type":"ContainerStarted","Data":"e5a40ce159af3efdfe5175e8b7835a7a77a90f03e3afebf9bfcc25f9daab88f3"} Aug 03 23:39:46.658736 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:46.658334 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdbb769bb-9vfpm" event={"ID":"031ccbfe-ab72-4869-82f1-076f05d4dcf4","Type":"ContainerStarted","Data":"442189421db8d010107de94dcde1537b8988c0484fb1eabaf0c98ab8f4195e98"} Aug 03 23:39:46.678778 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:46.678722 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bdbb769bb-9vfpm" podStartSLOduration=1.6787007840000001 podStartE2EDuration="1.678700784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:39:46.676202377 +0000 UTC m=+600.739458808" watchObservedRunningTime="2026-08-03 23:39:46.678700784 +0000 UTC m=+600.741957215" Aug 03 23:39:48.084327 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.084274 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t5t2j"] Aug 03 23:39:48.091408 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.091379 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:48.093929 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.093893 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 03 23:39:48.094078 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.093991 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-g6b5v\"" Aug 03 23:39:48.098173 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.098143 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t5t2j"] Aug 03 23:39:48.184421 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.184381 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc"] Aug 03 23:39:48.186851 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.186827 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.189260 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.189229 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 03 23:39:48.189429 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.189299 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 03 23:39:48.189514 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.189435 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7rc2k\"" Aug 03 23:39:48.189599 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.189575 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 03 23:39:48.189946 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.189931 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 03 23:39:48.190266 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.190251 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 03 23:39:48.198880 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.198847 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2f3444-26a7-41ca-986b-e808796cf632-cert\") pod \"odh-model-controller-7998c67dc4-t5t2j\" (UID: \"3b2f3444-26a7-41ca-986b-e808796cf632\") " pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:48.198990 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.198908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhtp\" (UniqueName: \"kubernetes.io/projected/3b2f3444-26a7-41ca-986b-e808796cf632-kube-api-access-skhtp\") pod \"odh-model-controller-7998c67dc4-t5t2j\" (UID: \"3b2f3444-26a7-41ca-986b-e808796cf632\") " pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:48.199816 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.199799 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc"] Aug 03 23:39:48.300033 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.300003 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e1292b8-1180-4bb0-836d-0ad18b629d4a-cert\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.300033 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.300038 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e1292b8-1180-4bb0-836d-0ad18b629d4a-manager-config\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.300249 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.300068 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhtp\" (UniqueName: \"kubernetes.io/projected/3b2f3444-26a7-41ca-986b-e808796cf632-kube-api-access-skhtp\") pod \"odh-model-controller-7998c67dc4-t5t2j\" (UID: \"3b2f3444-26a7-41ca-986b-e808796cf632\") " pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:48.300249 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.300188 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7gh\" (UniqueName: \"kubernetes.io/projected/6e1292b8-1180-4bb0-836d-0ad18b629d4a-kube-api-access-4w7gh\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.300249 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.300223 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2f3444-26a7-41ca-986b-e808796cf632-cert\") pod \"odh-model-controller-7998c67dc4-t5t2j\" (UID: \"3b2f3444-26a7-41ca-986b-e808796cf632\") " pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:48.300249 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.300241 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e1292b8-1180-4bb0-836d-0ad18b629d4a-metrics-cert\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.302662 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.302640 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2f3444-26a7-41ca-986b-e808796cf632-cert\") pod \"odh-model-controller-7998c67dc4-t5t2j\" (UID: \"3b2f3444-26a7-41ca-986b-e808796cf632\") " pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:48.310212 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.310186 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhtp\" (UniqueName: \"kubernetes.io/projected/3b2f3444-26a7-41ca-986b-e808796cf632-kube-api-access-skhtp\") pod \"odh-model-controller-7998c67dc4-t5t2j\" (UID: \"3b2f3444-26a7-41ca-986b-e808796cf632\") " pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:48.401282 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.401191 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7gh\" (UniqueName: \"kubernetes.io/projected/6e1292b8-1180-4bb0-836d-0ad18b629d4a-kube-api-access-4w7gh\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.401282 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.401239 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e1292b8-1180-4bb0-836d-0ad18b629d4a-metrics-cert\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.401282 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.401276 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e1292b8-1180-4bb0-836d-0ad18b629d4a-cert\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.401710 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.401332 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e1292b8-1180-4bb0-836d-0ad18b629d4a-manager-config\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.402005 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.401983 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e1292b8-1180-4bb0-836d-0ad18b629d4a-manager-config\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.404001 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.403981 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e1292b8-1180-4bb0-836d-0ad18b629d4a-metrics-cert\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.404062 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.404037 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e1292b8-1180-4bb0-836d-0ad18b629d4a-cert\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.405838 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.405822 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:48.409769 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.409740 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7gh\" (UniqueName: \"kubernetes.io/projected/6e1292b8-1180-4bb0-836d-0ad18b629d4a-kube-api-access-4w7gh\") pod \"lws-controller-manager-d64bf5b4d-jv4dc\" (UID: \"6e1292b8-1180-4bb0-836d-0ad18b629d4a\") " pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.496926 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.496868 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:48.545124 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.545098 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t5t2j"] Aug 03 23:39:48.547576 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:39:48.547545 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2f3444_26a7_41ca_986b_e808796cf632.slice/crio-2dcf102cbde453b6a5a0030099457220ec1bb3f364efa4a13e639c8fc2884b92 WatchSource:0}: Error finding container 2dcf102cbde453b6a5a0030099457220ec1bb3f364efa4a13e639c8fc2884b92: Status 404 returned error can't find the container with id 2dcf102cbde453b6a5a0030099457220ec1bb3f364efa4a13e639c8fc2884b92 Aug 03 23:39:48.646246 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.646224 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc"] Aug 03 23:39:48.649085 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:39:48.649057 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1292b8_1180_4bb0_836d_0ad18b629d4a.slice/crio-641de64b75213724bf19e1228e88319872213ff2ca7c494cd5e80d678f6a2a72 WatchSource:0}: Error finding container 641de64b75213724bf19e1228e88319872213ff2ca7c494cd5e80d678f6a2a72: Status 404 returned error can't find the container with id 641de64b75213724bf19e1228e88319872213ff2ca7c494cd5e80d678f6a2a72 Aug 03 23:39:48.668250 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.668223 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" event={"ID":"3b2f3444-26a7-41ca-986b-e808796cf632","Type":"ContainerStarted","Data":"2dcf102cbde453b6a5a0030099457220ec1bb3f364efa4a13e639c8fc2884b92"} Aug 03 23:39:48.669820 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:48.669794 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" event={"ID":"6e1292b8-1180-4bb0-836d-0ad18b629d4a","Type":"ContainerStarted","Data":"641de64b75213724bf19e1228e88319872213ff2ca7c494cd5e80d678f6a2a72"} Aug 03 23:39:51.689464 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:51.689430 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" event={"ID":"6e1292b8-1180-4bb0-836d-0ad18b629d4a","Type":"ContainerStarted","Data":"fb3b9a1bfc63b9ec831b1cad8c7397fa49ca38b9731295c45eedebb18adea6b6"} Aug 03 23:39:51.689884 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:51.689490 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:39:51.690890 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:51.690866 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" event={"ID":"3b2f3444-26a7-41ca-986b-e808796cf632","Type":"ContainerStarted","Data":"e88130a272f79653f119ec22d5275a4e7e7971c2a5ad96925757603c747da72b"} Aug 03 23:39:51.690997 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:51.690917 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:39:51.710941 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:51.708342 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" podStartSLOduration=0.895417068 podStartE2EDuration="3.708301048s" totalImagesPullingTime="2.81288398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:39:51.705438521 +0000 UTC m=+605.768694952" watchObservedRunningTime="2026-08-03 23:39:51.708301048 +0000 UTC m=+605.771557480" Aug 03 23:39:51.724581 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:51.724494 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" podStartSLOduration=0.763659137 podStartE2EDuration="3.724479055s" totalImagesPullingTime="2.960819918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:39:51.72326291 +0000 UTC m=+605.786519345" watchObservedRunningTime="2026-08-03 23:39:51.724479055 +0000 UTC m=+605.787735484" Aug 03 23:39:56.146121 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.146088 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:56.146532 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.146149 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:56.151031 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.151008 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:56.718252 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.718219 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bdbb769bb-9vfpm" Aug 03 23:39:56.771332 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.770403 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4c79b898-9krwv"] Aug 03 23:39:56.968994 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.968914 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm"] Aug 03 23:39:56.972497 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.972478 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:56.975355 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.975330 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 03 23:39:56.975473 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.975368 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 03 23:39:56.975473 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.975380 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 03 23:39:56.975473 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.975390 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2k6w9\"" Aug 03 23:39:56.975601 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.975589 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 03 23:39:56.975719 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.975690 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 03 23:39:56.975816 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.975771 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 03 23:39:56.986415 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:56.986389 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm"] Aug 03 23:39:57.100158 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.100121 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4546db1-5f37-4b78-a0e9-37f65ece8d40-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.100350 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.100165 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.100350 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.100194 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.100350 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.100218 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.100350 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.100328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9js7\" (UniqueName: \"kubernetes.io/projected/d4546db1-5f37-4b78-a0e9-37f65ece8d40-kube-api-access-t9js7\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.100484 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.100366 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.100484 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.100399 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.201162 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.201126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4546db1-5f37-4b78-a0e9-37f65ece8d40-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.201631 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.201177 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.201631 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.201198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.201631 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.201214 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.201631 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.201257 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9js7\" (UniqueName: \"kubernetes.io/projected/d4546db1-5f37-4b78-a0e9-37f65ece8d40-kube-api-access-t9js7\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.201631 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.201285 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.201631 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.201351 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.201939 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.201872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.203730 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.203705 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.203824 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.203808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.204145 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.204124 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4546db1-5f37-4b78-a0e9-37f65ece8d40-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.204687 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.204668 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.209501 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.209477 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4546db1-5f37-4b78-a0e9-37f65ece8d40-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.209780 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.209765 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9js7\" (UniqueName: \"kubernetes.io/projected/d4546db1-5f37-4b78-a0e9-37f65ece8d40-kube-api-access-t9js7\") pod \"istiod-openshift-gateway-5677ccb49b-qbbfm\" (UID: \"d4546db1-5f37-4b78-a0e9-37f65ece8d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.281990 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.281935 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:39:57.433531 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.433503 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm"] Aug 03 23:39:57.435791 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:39:57.435756 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4546db1_5f37_4b78_a0e9_37f65ece8d40.slice/crio-e7ab783e3c04291d969ab86a53eaa7111708cb7db48ac2cba7aaddab2c864cbc WatchSource:0}: Error finding container e7ab783e3c04291d969ab86a53eaa7111708cb7db48ac2cba7aaddab2c864cbc: Status 404 returned error can't find the container with id e7ab783e3c04291d969ab86a53eaa7111708cb7db48ac2cba7aaddab2c864cbc Aug 03 23:39:57.437878 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.437860 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 03 23:39:57.721666 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:57.721627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" event={"ID":"d4546db1-5f37-4b78-a0e9-37f65ece8d40","Type":"ContainerStarted","Data":"e7ab783e3c04291d969ab86a53eaa7111708cb7db48ac2cba7aaddab2c864cbc"} Aug 03 23:39:59.914486 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:59.914450 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 03 23:39:59.914805 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:39:59.914523 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 03 23:40:00.742753 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:00.742712 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" event={"ID":"d4546db1-5f37-4b78-a0e9-37f65ece8d40","Type":"ContainerStarted","Data":"34f93a96f4a6d664f246e4afc061a6acc47a8f2599c4d405e94b8954a97c27f0"} Aug 03 23:40:00.742910 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:00.742772 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:40:00.744474 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:00.744440 2556 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-qbbfm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 03 23:40:00.744620 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:00.744502 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" podUID="d4546db1-5f37-4b78-a0e9-37f65ece8d40" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 03 23:40:00.764530 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:00.764487 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" podStartSLOduration=2.288336386 podStartE2EDuration="4.764472177s" totalImagesPullingTime="2.476135791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:40:00.762148147 +0000 UTC m=+614.825404590" watchObservedRunningTime="2026-08-03 23:40:00.764472177 +0000 UTC m=+614.827728607" Aug 03 23:40:01.748559 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:01.748527 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qbbfm" Aug 03 23:40:02.698127 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:02.698092 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d64bf5b4d-jv4dc" Aug 03 23:40:02.698331 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:02.698155 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-t5t2j" Aug 03 23:40:21.801475 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:21.801422 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c4c79b898-9krwv" podUID="a913c316-f6ec-427f-aca8-890f9318e20f" containerName="console" containerID="cri-o://b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d" gracePeriod=15 Aug 03 23:40:22.043795 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.043772 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:40:22.119090 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119025 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-serving-cert\") pod \"a913c316-f6ec-427f-aca8-890f9318e20f\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " Aug 03 23:40:22.119090 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119068 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-service-ca\") pod \"a913c316-f6ec-427f-aca8-890f9318e20f\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " Aug 03 23:40:22.119273 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119098 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-oauth-config\") pod \"a913c316-f6ec-427f-aca8-890f9318e20f\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " Aug 03 23:40:22.119273 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119123 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6nt7\" (UniqueName: \"kubernetes.io/projected/a913c316-f6ec-427f-aca8-890f9318e20f-kube-api-access-c6nt7\") pod \"a913c316-f6ec-427f-aca8-890f9318e20f\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " Aug 03 23:40:22.119273 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119157 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-oauth-serving-cert\") pod \"a913c316-f6ec-427f-aca8-890f9318e20f\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " Aug 03 23:40:22.119273 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119200 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-console-config\") pod \"a913c316-f6ec-427f-aca8-890f9318e20f\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " Aug 03 23:40:22.119273 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119247 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-trusted-ca-bundle\") pod \"a913c316-f6ec-427f-aca8-890f9318e20f\" (UID: \"a913c316-f6ec-427f-aca8-890f9318e20f\") " Aug 03 23:40:22.119562 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119514 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-service-ca" (OuterVolumeSpecName: "service-ca") pod "a913c316-f6ec-427f-aca8-890f9318e20f" (UID: "a913c316-f6ec-427f-aca8-890f9318e20f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:40:22.119628 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119596 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:22.119628 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119585 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a913c316-f6ec-427f-aca8-890f9318e20f" (UID: "a913c316-f6ec-427f-aca8-890f9318e20f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:40:22.119628 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119606 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-console-config" (OuterVolumeSpecName: "console-config") pod "a913c316-f6ec-427f-aca8-890f9318e20f" (UID: "a913c316-f6ec-427f-aca8-890f9318e20f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:40:22.119931 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.119905 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a913c316-f6ec-427f-aca8-890f9318e20f" (UID: "a913c316-f6ec-427f-aca8-890f9318e20f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 03 23:40:22.121549 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.121524 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a913c316-f6ec-427f-aca8-890f9318e20f-kube-api-access-c6nt7" (OuterVolumeSpecName: "kube-api-access-c6nt7") pod "a913c316-f6ec-427f-aca8-890f9318e20f" (UID: "a913c316-f6ec-427f-aca8-890f9318e20f"). InnerVolumeSpecName "kube-api-access-c6nt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:40:22.121799 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.121779 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a913c316-f6ec-427f-aca8-890f9318e20f" (UID: "a913c316-f6ec-427f-aca8-890f9318e20f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 03 23:40:22.121877 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.121800 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a913c316-f6ec-427f-aca8-890f9318e20f" (UID: "a913c316-f6ec-427f-aca8-890f9318e20f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 03 23:40:22.220932 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.220906 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:22.220932 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.220930 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a913c316-f6ec-427f-aca8-890f9318e20f-console-oauth-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:22.221084 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.220940 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nt7\" (UniqueName: \"kubernetes.io/projected/a913c316-f6ec-427f-aca8-890f9318e20f-kube-api-access-c6nt7\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:22.221084 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.220951 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-oauth-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:22.221084 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.220959 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-console-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:22.221084 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.220967 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a913c316-f6ec-427f-aca8-890f9318e20f-trusted-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:22.853231 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.853199 2556 generic.go:356] "Generic (PLEG): container finished" podID="a913c316-f6ec-427f-aca8-890f9318e20f" containerID="b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d" exitCode=0 Aug 03 23:40:22.853699 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.853251 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4c79b898-9krwv" Aug 03 23:40:22.853699 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.853283 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4c79b898-9krwv" event={"ID":"a913c316-f6ec-427f-aca8-890f9318e20f","Type":"ContainerDied","Data":"b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d"} Aug 03 23:40:22.853699 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.853349 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4c79b898-9krwv" event={"ID":"a913c316-f6ec-427f-aca8-890f9318e20f","Type":"ContainerDied","Data":"3e5256a1f239656d76682dd336355ae83cbd09f8716a6d0e4a7e4f6b30c8203e"} Aug 03 23:40:22.853699 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.853368 2556 scope.go:117] "RemoveContainer" containerID="b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d" Aug 03 23:40:22.862045 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.862030 2556 scope.go:117] "RemoveContainer" containerID="b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d" Aug 03 23:40:22.862285 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:40:22.862268 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d\": container with ID starting with b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d not found: ID does not exist" containerID="b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d" Aug 03 23:40:22.862353 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.862291 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d"} err="failed to get container status \"b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d\": rpc error: code = NotFound desc = could not find container \"b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d\": container with ID starting with b876aa6056572a9ad77a7e1df0de5ffac59fd01dde9e880938edc1de1292c45d not found: ID does not exist" Aug 03 23:40:22.871588 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.871565 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4c79b898-9krwv"] Aug 03 23:40:22.876879 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:22.876859 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c4c79b898-9krwv"] Aug 03 23:40:24.420820 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:24.420783 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a913c316-f6ec-427f-aca8-890f9318e20f" path="/var/lib/kubelet/pods/a913c316-f6ec-427f-aca8-890f9318e20f/volumes" Aug 03 23:40:51.714359 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.714318 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4"] Aug 03 23:40:51.714721 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.714660 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a913c316-f6ec-427f-aca8-890f9318e20f" containerName="console" Aug 03 23:40:51.714721 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.714670 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a913c316-f6ec-427f-aca8-890f9318e20f" containerName="console" Aug 03 23:40:51.714810 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.714732 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a913c316-f6ec-427f-aca8-890f9318e20f" containerName="console" Aug 03 23:40:51.718973 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.718957 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.721689 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.721666 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fr74m\"" Aug 03 23:40:51.721811 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.721700 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 03 23:40:51.722663 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.722646 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 03 23:40:51.725895 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.725874 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4"] Aug 03 23:40:51.811530 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.811498 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx"] Aug 03 23:40:51.814967 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.814949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:51.823338 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.823296 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx"] Aug 03 23:40:51.866453 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.866429 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.866553 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.866460 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rbd\" (UniqueName: \"kubernetes.io/projected/cec873c4-0908-4024-9d55-f610d460878f-kube-api-access-f5rbd\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.866595 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.866563 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.905805 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.905778 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x"] Aug 03 23:40:51.909424 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.909409 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:51.915281 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.915261 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x"] Aug 03 23:40:51.967177 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.967113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.967177 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.967155 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:51.967455 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.967198 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:51.967455 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.967215 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.967455 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.967234 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rbd\" (UniqueName: \"kubernetes.io/projected/cec873c4-0908-4024-9d55-f610d460878f-kube-api-access-f5rbd\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.967455 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.967360 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qc2\" (UniqueName: \"kubernetes.io/projected/6dc6dce6-55ce-4256-ac76-0e837da9dad1-kube-api-access-l8qc2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:51.967652 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.967512 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.967652 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.967548 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:51.975511 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:51.975489 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rbd\" (UniqueName: \"kubernetes.io/projected/cec873c4-0908-4024-9d55-f610d460878f-kube-api-access-f5rbd\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:52.010147 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.010125 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9"] Aug 03 23:40:52.013820 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.013805 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.020800 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.020773 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9"] Aug 03 23:40:52.029047 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.029023 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:52.068017 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.067884 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:52.068017 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.067963 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.068017 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.068014 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:52.068288 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.068051 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.068288 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.068107 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qc2\" (UniqueName: \"kubernetes.io/projected/6dc6dce6-55ce-4256-ac76-0e837da9dad1-kube-api-access-l8qc2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:52.068288 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.068136 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9lkr\" (UniqueName: \"kubernetes.io/projected/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-kube-api-access-c9lkr\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.068692 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.068601 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:52.068852 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.068827 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:52.078145 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.078079 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qc2\" (UniqueName: \"kubernetes.io/projected/6dc6dce6-55ce-4256-ac76-0e837da9dad1-kube-api-access-l8qc2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:52.124713 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.124642 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:52.166191 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.166164 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4"] Aug 03 23:40:52.167787 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:40:52.167763 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec873c4_0908_4024_9d55_f610d460878f.slice/crio-2520cb0e1e182be6d834b403d31084b0fb768fe2f2efbf240ce8cad3cc58078c WatchSource:0}: Error finding container 2520cb0e1e182be6d834b403d31084b0fb768fe2f2efbf240ce8cad3cc58078c: Status 404 returned error can't find the container with id 2520cb0e1e182be6d834b403d31084b0fb768fe2f2efbf240ce8cad3cc58078c Aug 03 23:40:52.168758 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.168728 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.168873 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.168798 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.168873 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.168848 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.169024 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.168999 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.169083 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.169048 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9lkr\" (UniqueName: \"kubernetes.io/projected/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-kube-api-access-c9lkr\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.169150 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.169133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.169202 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.169187 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkzhp\" (UniqueName: \"kubernetes.io/projected/b7078e32-4c0b-41e2-909c-242a0d822745-kube-api-access-lkzhp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.169251 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.169201 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.177530 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.177509 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9lkr\" (UniqueName: \"kubernetes.io/projected/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-kube-api-access-c9lkr\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.219650 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.219621 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:52.270545 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.270510 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.270723 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.270567 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkzhp\" (UniqueName: \"kubernetes.io/projected/b7078e32-4c0b-41e2-909c-242a0d822745-kube-api-access-lkzhp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.270723 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.270606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.271183 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.270954 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.271183 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.271001 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.271183 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.271090 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx"] Aug 03 23:40:52.273877 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:40:52.273852 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc6dce6_55ce_4256_ac76_0e837da9dad1.slice/crio-7e2ffcace55d825dca1139447d158422c0abd584a7edb3e0eef46cb92a3bd542 WatchSource:0}: Error finding container 7e2ffcace55d825dca1139447d158422c0abd584a7edb3e0eef46cb92a3bd542: Status 404 returned error can't find the container with id 7e2ffcace55d825dca1139447d158422c0abd584a7edb3e0eef46cb92a3bd542 Aug 03 23:40:52.280239 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.279571 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkzhp\" (UniqueName: \"kubernetes.io/projected/b7078e32-4c0b-41e2-909c-242a0d822745-kube-api-access-lkzhp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.324871 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.324836 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:52.366949 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.366924 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x"] Aug 03 23:40:52.369404 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:40:52.369370 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e867de7_5e43_4a99_a3e9_5bf4a8f3d1fb.slice/crio-9012e8df2d5dc6b16f5f6ea7cf0e33f7f7cae1122957bea3a85710788f5921cc WatchSource:0}: Error finding container 9012e8df2d5dc6b16f5f6ea7cf0e33f7f7cae1122957bea3a85710788f5921cc: Status 404 returned error can't find the container with id 9012e8df2d5dc6b16f5f6ea7cf0e33f7f7cae1122957bea3a85710788f5921cc Aug 03 23:40:52.476094 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.476067 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9"] Aug 03 23:40:52.487322 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:40:52.487279 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7078e32_4c0b_41e2_909c_242a0d822745.slice/crio-b531291d80a5691e663b66057ca2381b21bb35b4570e4c3a9658000064a886a0 WatchSource:0}: Error finding container b531291d80a5691e663b66057ca2381b21bb35b4570e4c3a9658000064a886a0: Status 404 returned error can't find the container with id b531291d80a5691e663b66057ca2381b21bb35b4570e4c3a9658000064a886a0 Aug 03 23:40:52.996609 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.996579 2556 generic.go:356] "Generic (PLEG): container finished" podID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerID="1f024ecdb2fdf7d804188916c857d1f0aa3f2903ea4ad59af7d4a94614047e35" exitCode=0 Aug 03 23:40:52.997159 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.996616 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" event={"ID":"6dc6dce6-55ce-4256-ac76-0e837da9dad1","Type":"ContainerDied","Data":"1f024ecdb2fdf7d804188916c857d1f0aa3f2903ea4ad59af7d4a94614047e35"} Aug 03 23:40:52.997159 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.996677 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" event={"ID":"6dc6dce6-55ce-4256-ac76-0e837da9dad1","Type":"ContainerStarted","Data":"7e2ffcace55d825dca1139447d158422c0abd584a7edb3e0eef46cb92a3bd542"} Aug 03 23:40:52.998466 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.998440 2556 generic.go:356] "Generic (PLEG): container finished" podID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerID="8d6c8627d767054e05bd3393d7dcbef94f8e804d0e2b704a67e9d30f800c9f86" exitCode=0 Aug 03 23:40:52.998573 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.998523 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" event={"ID":"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb","Type":"ContainerDied","Data":"8d6c8627d767054e05bd3393d7dcbef94f8e804d0e2b704a67e9d30f800c9f86"} Aug 03 23:40:52.998573 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:52.998566 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" event={"ID":"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb","Type":"ContainerStarted","Data":"9012e8df2d5dc6b16f5f6ea7cf0e33f7f7cae1122957bea3a85710788f5921cc"} Aug 03 23:40:53.000209 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:53.000187 2556 generic.go:356] "Generic (PLEG): container finished" podID="b7078e32-4c0b-41e2-909c-242a0d822745" containerID="a9d65f795edf5cc8d283b2abeb6cdbd2f071861ba9e527353f4fa4c244daaac4" exitCode=0 Aug 03 23:40:53.000328 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:53.000276 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" event={"ID":"b7078e32-4c0b-41e2-909c-242a0d822745","Type":"ContainerDied","Data":"a9d65f795edf5cc8d283b2abeb6cdbd2f071861ba9e527353f4fa4c244daaac4"} Aug 03 23:40:53.000405 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:53.000345 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" event={"ID":"b7078e32-4c0b-41e2-909c-242a0d822745","Type":"ContainerStarted","Data":"b531291d80a5691e663b66057ca2381b21bb35b4570e4c3a9658000064a886a0"} Aug 03 23:40:53.002039 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:53.002015 2556 generic.go:356] "Generic (PLEG): container finished" podID="cec873c4-0908-4024-9d55-f610d460878f" containerID="fc9211934318a138118e528a08f644b9e85b7eda38ab06d055c72921cbbc76a6" exitCode=0 Aug 03 23:40:53.002118 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:53.002044 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" event={"ID":"cec873c4-0908-4024-9d55-f610d460878f","Type":"ContainerDied","Data":"fc9211934318a138118e528a08f644b9e85b7eda38ab06d055c72921cbbc76a6"} Aug 03 23:40:53.002118 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:53.002081 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" event={"ID":"cec873c4-0908-4024-9d55-f610d460878f","Type":"ContainerStarted","Data":"2520cb0e1e182be6d834b403d31084b0fb768fe2f2efbf240ce8cad3cc58078c"} Aug 03 23:40:54.009212 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:54.009186 2556 generic.go:356] "Generic (PLEG): container finished" podID="b7078e32-4c0b-41e2-909c-242a0d822745" containerID="0246f741b7b6c5d9885effe138db7c21320675f8308b08ec0cc730a875f837fb" exitCode=0 Aug 03 23:40:54.009608 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:54.009258 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" event={"ID":"b7078e32-4c0b-41e2-909c-242a0d822745","Type":"ContainerDied","Data":"0246f741b7b6c5d9885effe138db7c21320675f8308b08ec0cc730a875f837fb"} Aug 03 23:40:54.011443 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:54.011416 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" event={"ID":"cec873c4-0908-4024-9d55-f610d460878f","Type":"ContainerDied","Data":"ffed4c59159d47a1a142d0d24b8b278c50b292177a1a30650bfaf3a2f8fc3c4e"} Aug 03 23:40:54.011568 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:54.011529 2556 generic.go:356] "Generic (PLEG): container finished" podID="cec873c4-0908-4024-9d55-f610d460878f" containerID="ffed4c59159d47a1a142d0d24b8b278c50b292177a1a30650bfaf3a2f8fc3c4e" exitCode=0 Aug 03 23:40:55.018730 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:55.018697 2556 generic.go:356] "Generic (PLEG): container finished" podID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerID="5b94a57b994f6d6ba19aaf5c284f842630a3793aa8d9770214c791eff59dd0dd" exitCode=0 Aug 03 23:40:55.019129 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:55.018781 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" event={"ID":"6dc6dce6-55ce-4256-ac76-0e837da9dad1","Type":"ContainerDied","Data":"5b94a57b994f6d6ba19aaf5c284f842630a3793aa8d9770214c791eff59dd0dd"} Aug 03 23:40:55.020927 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:55.020907 2556 generic.go:356] "Generic (PLEG): container finished" podID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerID="845a8b20b88c0a577bbaa8bd3f9aeb09f77355eeff10b04ab886ff4ad07a00b3" exitCode=0 Aug 03 23:40:55.021049 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:55.021024 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" event={"ID":"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb","Type":"ContainerDied","Data":"845a8b20b88c0a577bbaa8bd3f9aeb09f77355eeff10b04ab886ff4ad07a00b3"} Aug 03 23:40:55.023531 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:55.023511 2556 generic.go:356] "Generic (PLEG): container finished" podID="b7078e32-4c0b-41e2-909c-242a0d822745" containerID="a6a7a01851982b340cb6fdc80fcbf9e570d687210ad7181dcc4b44980ec93d25" exitCode=0 Aug 03 23:40:55.023618 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:55.023558 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" event={"ID":"b7078e32-4c0b-41e2-909c-242a0d822745","Type":"ContainerDied","Data":"a6a7a01851982b340cb6fdc80fcbf9e570d687210ad7181dcc4b44980ec93d25"} Aug 03 23:40:55.025777 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:55.025757 2556 generic.go:356] "Generic (PLEG): container finished" podID="cec873c4-0908-4024-9d55-f610d460878f" containerID="aabe32fba21b31f4246a0eaca862c23270ea8599ad484b9cdf46295034aca62c" exitCode=0 Aug 03 23:40:55.025865 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:55.025809 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" event={"ID":"cec873c4-0908-4024-9d55-f610d460878f","Type":"ContainerDied","Data":"aabe32fba21b31f4246a0eaca862c23270ea8599ad484b9cdf46295034aca62c"} Aug 03 23:40:56.032703 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.032666 2556 generic.go:356] "Generic (PLEG): container finished" podID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerID="4e20d13987f02ef34b6891fd7f230f298cc2b1016488437861d19799c5844e16" exitCode=0 Aug 03 23:40:56.033134 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.032752 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" event={"ID":"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb","Type":"ContainerDied","Data":"4e20d13987f02ef34b6891fd7f230f298cc2b1016488437861d19799c5844e16"} Aug 03 23:40:56.034851 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.034828 2556 generic.go:356] "Generic (PLEG): container finished" podID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerID="1d5709c7c753f1fe2cb4c71f649e59ea824f8e09a129414588a8ea3f6d3733f5" exitCode=0 Aug 03 23:40:56.034966 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.034908 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" event={"ID":"6dc6dce6-55ce-4256-ac76-0e837da9dad1","Type":"ContainerDied","Data":"1d5709c7c753f1fe2cb4c71f649e59ea824f8e09a129414588a8ea3f6d3733f5"} Aug 03 23:40:56.193882 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.193857 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:56.196896 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.196871 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:56.304856 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.304771 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-bundle\") pod \"cec873c4-0908-4024-9d55-f610d460878f\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " Aug 03 23:40:56.304856 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.304827 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-util\") pod \"b7078e32-4c0b-41e2-909c-242a0d822745\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " Aug 03 23:40:56.305074 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.304862 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-bundle\") pod \"b7078e32-4c0b-41e2-909c-242a0d822745\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " Aug 03 23:40:56.305074 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.304895 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkzhp\" (UniqueName: \"kubernetes.io/projected/b7078e32-4c0b-41e2-909c-242a0d822745-kube-api-access-lkzhp\") pod \"b7078e32-4c0b-41e2-909c-242a0d822745\" (UID: \"b7078e32-4c0b-41e2-909c-242a0d822745\") " Aug 03 23:40:56.305074 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.304921 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5rbd\" (UniqueName: \"kubernetes.io/projected/cec873c4-0908-4024-9d55-f610d460878f-kube-api-access-f5rbd\") pod \"cec873c4-0908-4024-9d55-f610d460878f\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " Aug 03 23:40:56.305074 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.304970 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-util\") pod \"cec873c4-0908-4024-9d55-f610d460878f\" (UID: \"cec873c4-0908-4024-9d55-f610d460878f\") " Aug 03 23:40:56.305500 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.305469 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-bundle" (OuterVolumeSpecName: "bundle") pod "b7078e32-4c0b-41e2-909c-242a0d822745" (UID: "b7078e32-4c0b-41e2-909c-242a0d822745"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:40:56.305642 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.305588 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-bundle" (OuterVolumeSpecName: "bundle") pod "cec873c4-0908-4024-9d55-f610d460878f" (UID: "cec873c4-0908-4024-9d55-f610d460878f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:40:56.307275 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.307243 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7078e32-4c0b-41e2-909c-242a0d822745-kube-api-access-lkzhp" (OuterVolumeSpecName: "kube-api-access-lkzhp") pod "b7078e32-4c0b-41e2-909c-242a0d822745" (UID: "b7078e32-4c0b-41e2-909c-242a0d822745"). InnerVolumeSpecName "kube-api-access-lkzhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:40:56.307404 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.307359 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec873c4-0908-4024-9d55-f610d460878f-kube-api-access-f5rbd" (OuterVolumeSpecName: "kube-api-access-f5rbd") pod "cec873c4-0908-4024-9d55-f610d460878f" (UID: "cec873c4-0908-4024-9d55-f610d460878f"). InnerVolumeSpecName "kube-api-access-f5rbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:40:56.312802 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.312777 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-util" (OuterVolumeSpecName: "util") pod "cec873c4-0908-4024-9d55-f610d460878f" (UID: "cec873c4-0908-4024-9d55-f610d460878f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:40:56.313750 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.313730 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-util" (OuterVolumeSpecName: "util") pod "b7078e32-4c0b-41e2-909c-242a0d822745" (UID: "b7078e32-4c0b-41e2-909c-242a0d822745"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:40:56.405760 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.405724 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:56.405760 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.405752 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-util\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:56.405760 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.405763 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7078e32-4c0b-41e2-909c-242a0d822745-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:56.405976 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.405771 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkzhp\" (UniqueName: \"kubernetes.io/projected/b7078e32-4c0b-41e2-909c-242a0d822745-kube-api-access-lkzhp\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:56.405976 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.405780 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rbd\" (UniqueName: \"kubernetes.io/projected/cec873c4-0908-4024-9d55-f610d460878f-kube-api-access-f5rbd\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:56.405976 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:56.405788 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cec873c4-0908-4024-9d55-f610d460878f-util\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:57.041966 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.041921 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" event={"ID":"b7078e32-4c0b-41e2-909c-242a0d822745","Type":"ContainerDied","Data":"b531291d80a5691e663b66057ca2381b21bb35b4570e4c3a9658000064a886a0"} Aug 03 23:40:57.041966 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.041948 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb9dns9" Aug 03 23:40:57.042525 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.042014 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b531291d80a5691e663b66057ca2381b21bb35b4570e4c3a9658000064a886a0" Aug 03 23:40:57.043849 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.043770 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" Aug 03 23:40:57.043849 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.043790 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870535vhx4" event={"ID":"cec873c4-0908-4024-9d55-f610d460878f","Type":"ContainerDied","Data":"2520cb0e1e182be6d834b403d31084b0fb768fe2f2efbf240ce8cad3cc58078c"} Aug 03 23:40:57.043849 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.043835 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2520cb0e1e182be6d834b403d31084b0fb768fe2f2efbf240ce8cad3cc58078c" Aug 03 23:40:57.199920 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.199898 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:57.238792 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.238768 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:57.313706 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.313626 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-util\") pod \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " Aug 03 23:40:57.313706 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.313701 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-bundle\") pod \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " Aug 03 23:40:57.313909 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.313744 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9lkr\" (UniqueName: \"kubernetes.io/projected/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-kube-api-access-c9lkr\") pod \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\" (UID: \"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb\") " Aug 03 23:40:57.314185 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.314149 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-bundle" (OuterVolumeSpecName: "bundle") pod "7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" (UID: "7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:40:57.316003 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.315981 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-kube-api-access-c9lkr" (OuterVolumeSpecName: "kube-api-access-c9lkr") pod "7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" (UID: "7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb"). InnerVolumeSpecName "kube-api-access-c9lkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:40:57.318792 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.318759 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-util" (OuterVolumeSpecName: "util") pod "7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" (UID: "7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:40:57.414855 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.414822 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-bundle\") pod \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " Aug 03 23:40:57.414855 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.414862 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8qc2\" (UniqueName: \"kubernetes.io/projected/6dc6dce6-55ce-4256-ac76-0e837da9dad1-kube-api-access-l8qc2\") pod \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " Aug 03 23:40:57.415087 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.414894 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-util\") pod \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\" (UID: \"6dc6dce6-55ce-4256-ac76-0e837da9dad1\") " Aug 03 23:40:57.415087 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.415080 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-util\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:57.415204 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.415091 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:57.415204 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.415100 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9lkr\" (UniqueName: \"kubernetes.io/projected/7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb-kube-api-access-c9lkr\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:57.415417 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.415387 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-bundle" (OuterVolumeSpecName: "bundle") pod "6dc6dce6-55ce-4256-ac76-0e837da9dad1" (UID: "6dc6dce6-55ce-4256-ac76-0e837da9dad1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:40:57.416977 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.416950 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc6dce6-55ce-4256-ac76-0e837da9dad1-kube-api-access-l8qc2" (OuterVolumeSpecName: "kube-api-access-l8qc2") pod "6dc6dce6-55ce-4256-ac76-0e837da9dad1" (UID: "6dc6dce6-55ce-4256-ac76-0e837da9dad1"). InnerVolumeSpecName "kube-api-access-l8qc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:40:57.420367 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.420346 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-util" (OuterVolumeSpecName: "util") pod "6dc6dce6-55ce-4256-ac76-0e837da9dad1" (UID: "6dc6dce6-55ce-4256-ac76-0e837da9dad1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 03 23:40:57.515694 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.515665 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:57.515694 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.515691 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qc2\" (UniqueName: \"kubernetes.io/projected/6dc6dce6-55ce-4256-ac76-0e837da9dad1-kube-api-access-l8qc2\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:57.515694 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:57.515701 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dc6dce6-55ce-4256-ac76-0e837da9dad1-util\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:40:58.050975 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:58.050946 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" Aug 03 23:40:58.051414 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:58.050970 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dthpx" event={"ID":"6dc6dce6-55ce-4256-ac76-0e837da9dad1","Type":"ContainerDied","Data":"7e2ffcace55d825dca1139447d158422c0abd584a7edb3e0eef46cb92a3bd542"} Aug 03 23:40:58.051414 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:58.051019 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2ffcace55d825dca1139447d158422c0abd584a7edb3e0eef46cb92a3bd542" Aug 03 23:40:58.052924 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:58.052903 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" Aug 03 23:40:58.053024 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:58.052909 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cdfc9x" event={"ID":"7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb","Type":"ContainerDied","Data":"9012e8df2d5dc6b16f5f6ea7cf0e33f7f7cae1122957bea3a85710788f5921cc"} Aug 03 23:40:58.053024 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:40:58.053017 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9012e8df2d5dc6b16f5f6ea7cf0e33f7f7cae1122957bea3a85710788f5921cc" Aug 03 23:41:09.405206 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405134 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg"] Aug 03 23:41:09.405723 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405705 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7078e32-4c0b-41e2-909c-242a0d822745" containerName="util" Aug 03 23:41:09.405771 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405730 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7078e32-4c0b-41e2-909c-242a0d822745" containerName="util" Aug 03 23:41:09.405771 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405752 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7078e32-4c0b-41e2-909c-242a0d822745" containerName="pull" Aug 03 23:41:09.405771 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405759 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7078e32-4c0b-41e2-909c-242a0d822745" containerName="pull" Aug 03 23:41:09.405771 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405767 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerName="pull" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405775 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerName="pull" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405789 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerName="extract" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405797 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerName="extract" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405810 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec873c4-0908-4024-9d55-f610d460878f" containerName="util" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405817 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec873c4-0908-4024-9d55-f610d460878f" containerName="util" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405824 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7078e32-4c0b-41e2-909c-242a0d822745" containerName="extract" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405830 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7078e32-4c0b-41e2-909c-242a0d822745" containerName="extract" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405840 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec873c4-0908-4024-9d55-f610d460878f" containerName="pull" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405845 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec873c4-0908-4024-9d55-f610d460878f" containerName="pull" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405850 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerName="pull" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405855 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerName="pull" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405860 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerName="extract" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405865 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerName="extract" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405870 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerName="util" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405875 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerName="util" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405879 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec873c4-0908-4024-9d55-f610d460878f" containerName="extract" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405884 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec873c4-0908-4024-9d55-f610d460878f" containerName="extract" Aug 03 23:41:09.405885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405891 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerName="util" Aug 03 23:41:09.406448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405896 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerName="util" Aug 03 23:41:09.406448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405981 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e867de7-5e43-4a99-a3e9-5bf4a8f3d1fb" containerName="extract" Aug 03 23:41:09.406448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405988 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7078e32-4c0b-41e2-909c-242a0d822745" containerName="extract" Aug 03 23:41:09.406448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.405994 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc6dce6-55ce-4256-ac76-0e837da9dad1" containerName="extract" Aug 03 23:41:09.406448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.406001 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec873c4-0908-4024-9d55-f610d460878f" containerName="extract" Aug 03 23:41:09.408995 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.408979 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:09.411766 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.411744 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tnnlr\"" Aug 03 23:41:09.411868 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.411745 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 03 23:41:09.412768 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.412751 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 03 23:41:09.414106 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.414087 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glv4t\" (UniqueName: \"kubernetes.io/projected/7daa7345-fdc3-4e8a-836c-296e5524929c-kube-api-access-glv4t\") pod \"limitador-operator-controller-manager-6c549b6d85-w2ntg\" (UID: \"7daa7345-fdc3-4e8a-836c-296e5524929c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:09.418538 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.418514 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg"] Aug 03 23:41:09.514522 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.514481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glv4t\" (UniqueName: \"kubernetes.io/projected/7daa7345-fdc3-4e8a-836c-296e5524929c-kube-api-access-glv4t\") pod \"limitador-operator-controller-manager-6c549b6d85-w2ntg\" (UID: \"7daa7345-fdc3-4e8a-836c-296e5524929c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:09.522936 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.522910 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glv4t\" (UniqueName: \"kubernetes.io/projected/7daa7345-fdc3-4e8a-836c-296e5524929c-kube-api-access-glv4t\") pod \"limitador-operator-controller-manager-6c549b6d85-w2ntg\" (UID: \"7daa7345-fdc3-4e8a-836c-296e5524929c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:09.720213 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.720184 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:09.857372 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:09.857342 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg"] Aug 03 23:41:09.858938 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:41:09.858910 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7daa7345_fdc3_4e8a_836c_296e5524929c.slice/crio-9a81878fbcccd56ee3b8d2f4dba1cbffb77b1fbf4f9c7fd8a2d8c8aec8312028 WatchSource:0}: Error finding container 9a81878fbcccd56ee3b8d2f4dba1cbffb77b1fbf4f9c7fd8a2d8c8aec8312028: Status 404 returned error can't find the container with id 9a81878fbcccd56ee3b8d2f4dba1cbffb77b1fbf4f9c7fd8a2d8c8aec8312028 Aug 03 23:41:10.113694 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:10.113605 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" event={"ID":"7daa7345-fdc3-4e8a-836c-296e5524929c","Type":"ContainerStarted","Data":"9a81878fbcccd56ee3b8d2f4dba1cbffb77b1fbf4f9c7fd8a2d8c8aec8312028"} Aug 03 23:41:13.131938 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:13.131897 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" event={"ID":"7daa7345-fdc3-4e8a-836c-296e5524929c","Type":"ContainerStarted","Data":"c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3"} Aug 03 23:41:13.132341 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:13.131973 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:13.150188 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:13.150141 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" podStartSLOduration=1.705178395 podStartE2EDuration="4.150126656s" totalImagesPullingTime="2.444948261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:41:13.147576282 +0000 UTC m=+687.210832711" watchObservedRunningTime="2026-08-03 23:41:13.150126656 +0000 UTC m=+687.213383086" Aug 03 23:41:23.687906 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.687874 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg"] Aug 03 23:41:23.688431 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.688095 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" containerName="manager" containerID="cri-o://c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3" gracePeriod=2 Aug 03 23:41:23.689942 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.689912 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:23.695465 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.695440 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg"] Aug 03 23:41:23.696254 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:41:23.696223 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-w2ntg\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:23.709643 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.709621 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v"] Aug 03 23:41:23.709965 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.709954 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" containerName="manager" Aug 03 23:41:23.710024 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.709966 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" containerName="manager" Aug 03 23:41:23.710061 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.710041 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" containerName="manager" Aug 03 23:41:23.713062 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.713047 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" Aug 03 23:41:23.715483 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:41:23.715458 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-w2ntg\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:23.724646 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.724625 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v"] Aug 03 23:41:23.838733 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.838706 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbh6\" (UniqueName: \"kubernetes.io/projected/85cee131-c0ac-464d-8b74-8957f00bee1b-kube-api-access-frbh6\") pod \"limitador-operator-controller-manager-6c549b6d85-b9m5v\" (UID: \"85cee131-c0ac-464d-8b74-8957f00bee1b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" Aug 03 23:41:23.928415 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.928388 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:23.931048 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:41:23.931019 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-w2ntg\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:23.939433 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.939390 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbh6\" (UniqueName: \"kubernetes.io/projected/85cee131-c0ac-464d-8b74-8957f00bee1b-kube-api-access-frbh6\") pod \"limitador-operator-controller-manager-6c549b6d85-b9m5v\" (UID: \"85cee131-c0ac-464d-8b74-8957f00bee1b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" Aug 03 23:41:23.953771 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:23.953748 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbh6\" (UniqueName: \"kubernetes.io/projected/85cee131-c0ac-464d-8b74-8957f00bee1b-kube-api-access-frbh6\") pod \"limitador-operator-controller-manager-6c549b6d85-b9m5v\" (UID: \"85cee131-c0ac-464d-8b74-8957f00bee1b\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" Aug 03 23:41:24.039843 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.039808 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glv4t\" (UniqueName: \"kubernetes.io/projected/7daa7345-fdc3-4e8a-836c-296e5524929c-kube-api-access-glv4t\") pod \"7daa7345-fdc3-4e8a-836c-296e5524929c\" (UID: \"7daa7345-fdc3-4e8a-836c-296e5524929c\") " Aug 03 23:41:24.042142 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.042117 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7daa7345-fdc3-4e8a-836c-296e5524929c-kube-api-access-glv4t" (OuterVolumeSpecName: "kube-api-access-glv4t") pod "7daa7345-fdc3-4e8a-836c-296e5524929c" (UID: "7daa7345-fdc3-4e8a-836c-296e5524929c"). InnerVolumeSpecName "kube-api-access-glv4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 03 23:41:24.075959 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.075932 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" Aug 03 23:41:24.141069 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.141031 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glv4t\" (UniqueName: \"kubernetes.io/projected/7daa7345-fdc3-4e8a-836c-296e5524929c-kube-api-access-glv4t\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 03 23:41:24.193961 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.193873 2556 generic.go:356] "Generic (PLEG): container finished" podID="7daa7345-fdc3-4e8a-836c-296e5524929c" containerID="c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3" exitCode=0 Aug 03 23:41:24.193961 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.193929 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:24.193961 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.193955 2556 scope.go:117] "RemoveContainer" containerID="c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3" Aug 03 23:41:24.196426 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:41:24.196380 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-w2ntg\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:24.204606 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.204579 2556 scope.go:117] "RemoveContainer" containerID="c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3" Aug 03 23:41:24.204980 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:41:24.204944 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3\": container with ID starting with c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3 not found: ID does not exist" containerID="c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3" Aug 03 23:41:24.205061 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.204997 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3"} err="failed to get container status \"c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3\": rpc error: code = NotFound desc = could not find container \"c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3\": container with ID starting with c7f1beae5e81531780ad0abe498ea0588b1f410c7c2e00e08d19381bbda920a3 not found: ID does not exist" Aug 03 23:41:24.209918 ip-10-0-136-126 kubenswrapper[2556]: E0803 23:41:24.209889 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-w2ntg\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-w2ntg" Aug 03 23:41:24.213615 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.213596 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v"] Aug 03 23:41:24.215152 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:41:24.215131 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cee131_c0ac_464d_8b74_8957f00bee1b.slice/crio-b0ac0b4e79535231e0d1ef9fb181a285578e1f38b53888b142d643555a69b311 WatchSource:0}: Error finding container b0ac0b4e79535231e0d1ef9fb181a285578e1f38b53888b142d643555a69b311: Status 404 returned error can't find the container with id b0ac0b4e79535231e0d1ef9fb181a285578e1f38b53888b142d643555a69b311 Aug 03 23:41:24.421729 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:24.421689 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7daa7345-fdc3-4e8a-836c-296e5524929c" path="/var/lib/kubelet/pods/7daa7345-fdc3-4e8a-836c-296e5524929c/volumes" Aug 03 23:41:25.201512 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:25.201472 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" event={"ID":"85cee131-c0ac-464d-8b74-8957f00bee1b","Type":"ContainerStarted","Data":"e877475b4936fde31495e0d2b3216d64cda6189b967dda437862a9b5877a3a75"} Aug 03 23:41:25.201921 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:25.201527 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" event={"ID":"85cee131-c0ac-464d-8b74-8957f00bee1b","Type":"ContainerStarted","Data":"b0ac0b4e79535231e0d1ef9fb181a285578e1f38b53888b142d643555a69b311"} Aug 03 23:41:25.201921 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:25.201630 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" Aug 03 23:41:25.218892 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:25.218845 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" podStartSLOduration=2.218830934 podStartE2EDuration="2.218830934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:41:25.216679681 +0000 UTC m=+699.279936114" watchObservedRunningTime="2026-08-03 23:41:25.218830934 +0000 UTC m=+699.282087364" Aug 03 23:41:36.209287 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:41:36.209251 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-b9m5v" Aug 03 23:42:11.704066 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.704036 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4"] Aug 03 23:42:11.707540 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.707522 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:11.710002 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.709982 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 03 23:42:11.711013 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.710994 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 03 23:42:11.711073 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.711018 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Aug 03 23:42:11.711073 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.711036 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vfz9s\"" Aug 03 23:42:11.716342 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.716285 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4"] Aug 03 23:42:11.727727 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.727704 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pq6m\" (UniqueName: \"kubernetes.io/projected/893acca7-29d3-4d1f-9753-eaad035d1b45-kube-api-access-8pq6m\") pod \"postgres-69fb4594cf-q4pv4\" (UID: \"893acca7-29d3-4d1f-9753-eaad035d1b45\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:11.727818 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.727779 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/893acca7-29d3-4d1f-9753-eaad035d1b45-data\") pod \"postgres-69fb4594cf-q4pv4\" (UID: \"893acca7-29d3-4d1f-9753-eaad035d1b45\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:11.828713 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.828677 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq6m\" (UniqueName: \"kubernetes.io/projected/893acca7-29d3-4d1f-9753-eaad035d1b45-kube-api-access-8pq6m\") pod \"postgres-69fb4594cf-q4pv4\" (UID: \"893acca7-29d3-4d1f-9753-eaad035d1b45\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:11.828883 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.828743 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/893acca7-29d3-4d1f-9753-eaad035d1b45-data\") pod \"postgres-69fb4594cf-q4pv4\" (UID: \"893acca7-29d3-4d1f-9753-eaad035d1b45\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:11.829126 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.829107 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/893acca7-29d3-4d1f-9753-eaad035d1b45-data\") pod \"postgres-69fb4594cf-q4pv4\" (UID: \"893acca7-29d3-4d1f-9753-eaad035d1b45\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:11.837492 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:11.837448 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq6m\" (UniqueName: \"kubernetes.io/projected/893acca7-29d3-4d1f-9753-eaad035d1b45-kube-api-access-8pq6m\") pod \"postgres-69fb4594cf-q4pv4\" (UID: \"893acca7-29d3-4d1f-9753-eaad035d1b45\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:12.018612 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:12.018581 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:12.152920 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:12.152898 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4"] Aug 03 23:42:12.155102 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:42:12.155076 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893acca7_29d3_4d1f_9753_eaad035d1b45.slice/crio-b2c415841383091cd9f57e79c45615311b1ab2a4cbc3100b5fce4aedeab7d839 WatchSource:0}: Error finding container b2c415841383091cd9f57e79c45615311b1ab2a4cbc3100b5fce4aedeab7d839: Status 404 returned error can't find the container with id b2c415841383091cd9f57e79c45615311b1ab2a4cbc3100b5fce4aedeab7d839 Aug 03 23:42:12.447525 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:12.447447 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" event={"ID":"893acca7-29d3-4d1f-9753-eaad035d1b45","Type":"ContainerStarted","Data":"b2c415841383091cd9f57e79c45615311b1ab2a4cbc3100b5fce4aedeab7d839"} Aug 03 23:42:17.481522 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:17.481492 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" event={"ID":"893acca7-29d3-4d1f-9753-eaad035d1b45","Type":"ContainerStarted","Data":"5c1da71856900b65cf8e285e3f087b183603a52dd8176956e85923ec5d34964a"} Aug 03 23:42:17.481913 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:17.481629 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:42:17.497380 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:17.497323 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" podStartSLOduration=1.24181747 podStartE2EDuration="6.497283362s" totalImagesPullingTime="5.255465892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:42:17.49475583 +0000 UTC m=+751.558012264" watchObservedRunningTime="2026-08-03 23:42:17.497283362 +0000 UTC m=+751.560539791" Aug 03 23:42:23.514743 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:42:23.514716 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-q4pv4" Aug 03 23:43:33.333551 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.333514 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc"] Aug 03 23:43:33.337751 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.337730 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.341646 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.341573 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 03 23:43:33.341917 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.341895 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 03 23:43:33.342063 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.341959 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 03 23:43:33.342137 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.341965 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jsgb6\"" Aug 03 23:43:33.343205 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.343186 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc"] Aug 03 23:43:33.443503 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.443473 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.443657 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.443508 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c880f829-6827-4943-bc05-2c53d40d1fd7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.443657 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.443613 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.443657 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.443637 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqxd\" (UniqueName: \"kubernetes.io/projected/c880f829-6827-4943-bc05-2c53d40d1fd7-kube-api-access-fhqxd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.443760 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.443668 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.443760 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.443705 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544102 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544076 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqxd\" (UniqueName: \"kubernetes.io/projected/c880f829-6827-4943-bc05-2c53d40d1fd7-kube-api-access-fhqxd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544102 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544108 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544372 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544372 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544372 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544362 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c880f829-6827-4943-bc05-2c53d40d1fd7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544530 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544581 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544525 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544667 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544641 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.544805 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.544786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.546397 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.546377 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c880f829-6827-4943-bc05-2c53d40d1fd7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.546707 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.546691 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c880f829-6827-4943-bc05-2c53d40d1fd7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.552518 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.552495 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqxd\" (UniqueName: \"kubernetes.io/projected/c880f829-6827-4943-bc05-2c53d40d1fd7-kube-api-access-fhqxd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc\" (UID: \"c880f829-6827-4943-bc05-2c53d40d1fd7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.650260 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.650183 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:33.780812 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.780789 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc"] Aug 03 23:43:33.782528 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:43:33.782498 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc880f829_6827_4943_bc05_2c53d40d1fd7.slice/crio-24b73231541ce3bb1a2092fda85950416bde46c01bc8bc8357bc209f5b2f9890 WatchSource:0}: Error finding container 24b73231541ce3bb1a2092fda85950416bde46c01bc8bc8357bc209f5b2f9890: Status 404 returned error can't find the container with id 24b73231541ce3bb1a2092fda85950416bde46c01bc8bc8357bc209f5b2f9890 Aug 03 23:43:33.860499 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:33.860467 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" event={"ID":"c880f829-6827-4943-bc05-2c53d40d1fd7","Type":"ContainerStarted","Data":"24b73231541ce3bb1a2092fda85950416bde46c01bc8bc8357bc209f5b2f9890"} Aug 03 23:43:35.871718 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:35.871684 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" event={"ID":"c880f829-6827-4943-bc05-2c53d40d1fd7","Type":"ContainerStarted","Data":"bfb34aa2745dcd59a7f7b2a1a9afaee81434f60109fd7f6beb9adf84ecc803d9"} Aug 03 23:43:35.890481 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:35.890427 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" podStartSLOduration=1.852935799 podStartE2EDuration="2.890411401s" totalImagesPullingTime="1.037475602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:43:35.887179151 +0000 UTC m=+829.950435581" watchObservedRunningTime="2026-08-03 23:43:35.890411401 +0000 UTC m=+829.953667829" Aug 03 23:43:43.650341 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:43.650288 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:43.650780 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:43.650397 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:43.662852 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:43.662824 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:43.928359 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:43.928262 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc" Aug 03 23:43:51.443167 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.443140 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw"] Aug 03 23:43:51.446656 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.446640 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.449204 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.449176 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 03 23:43:51.454532 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.454511 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw"] Aug 03 23:43:51.498281 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.498255 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.498448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.498297 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.498448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.498342 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkbd\" (UniqueName: \"kubernetes.io/projected/8a071b68-6494-4985-bd81-1974b2cd9cb7-kube-api-access-qgkbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.498448 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.498370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a071b68-6494-4985-bd81-1974b2cd9cb7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.498623 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.498443 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.498623 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.498489 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.599634 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.599600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.599784 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.599647 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.599784 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.599666 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkbd\" (UniqueName: \"kubernetes.io/projected/8a071b68-6494-4985-bd81-1974b2cd9cb7-kube-api-access-qgkbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.599872 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.599838 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a071b68-6494-4985-bd81-1974b2cd9cb7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.599921 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.599887 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.599921 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.599912 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.600018 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.599952 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.600072 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.600000 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.600172 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.600149 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.602191 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.602168 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a071b68-6494-4985-bd81-1974b2cd9cb7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.602343 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.602328 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a071b68-6494-4985-bd81-1974b2cd9cb7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.607941 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.607924 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkbd\" (UniqueName: \"kubernetes.io/projected/8a071b68-6494-4985-bd81-1974b2cd9cb7-kube-api-access-qgkbd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw\" (UID: \"8a071b68-6494-4985-bd81-1974b2cd9cb7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.758668 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.758642 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:43:51.887557 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.887530 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw"] Aug 03 23:43:51.888667 ip-10-0-136-126 kubenswrapper[2556]: W0803 23:43:51.888644 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a071b68_6494_4985_bd81_1974b2cd9cb7.slice/crio-6bdda9242a2c1fae93faafef8e6b19e93fdc1444cecba1a86a7f65e7b33f0b26 WatchSource:0}: Error finding container 6bdda9242a2c1fae93faafef8e6b19e93fdc1444cecba1a86a7f65e7b33f0b26: Status 404 returned error can't find the container with id 6bdda9242a2c1fae93faafef8e6b19e93fdc1444cecba1a86a7f65e7b33f0b26 Aug 03 23:43:51.957771 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:51.957736 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" event={"ID":"8a071b68-6494-4985-bd81-1974b2cd9cb7","Type":"ContainerStarted","Data":"6bdda9242a2c1fae93faafef8e6b19e93fdc1444cecba1a86a7f65e7b33f0b26"} Aug 03 23:43:52.963868 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:52.963829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" event={"ID":"8a071b68-6494-4985-bd81-1974b2cd9cb7","Type":"ContainerStarted","Data":"4357fb98cf7825fef890c4c0b48081c850146093024879de0c5c29bfdcf4fdd3"} Aug 03 23:43:52.979525 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:43:52.979465 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" podStartSLOduration=1.806059477 podStartE2EDuration="1.979448062s" totalImagesPullingTime="173.388585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-03 23:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-03 23:43:52.979242393 +0000 UTC m=+847.042498820" watchObservedRunningTime="2026-08-03 23:43:52.979448062 +0000 UTC m=+847.042704493" Aug 03 23:44:01.758865 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:44:01.758833 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:44:01.759232 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:44:01.758898 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:44:01.770947 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:44:01.770922 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:44:02.024618 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:44:02.024542 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw" Aug 03 23:44:46.598022 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:44:46.597905 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:44:46.604178 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:44:46.598422 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:49:46.633410 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:49:46.633293 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:49:46.642926 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:49:46.642041 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:54:46.676453 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:54:46.676347 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:54:46.680584 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:54:46.680180 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:59:46.715430 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:59:46.715398 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 03 23:59:46.719885 ip-10-0-136-126 kubenswrapper[2556]: I0803 23:59:46.719855 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:03:24.018530 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.018490 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2"] Aug 04 00:03:24.022575 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.022544 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" Aug 04 00:03:24.025728 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.025700 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:03:24.030565 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.030532 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2"] Aug 04 00:03:24.125341 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.125272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de86e5b-857e-42e8-8948-2f83d1784d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2\" (UID: \"3de86e5b-857e-42e8-8948-2f83d1784d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" Aug 04 00:03:24.226663 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.226626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de86e5b-857e-42e8-8948-2f83d1784d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2\" (UID: \"3de86e5b-857e-42e8-8948-2f83d1784d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" Aug 04 00:03:24.227296 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.227271 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de86e5b-857e-42e8-8948-2f83d1784d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2\" (UID: \"3de86e5b-857e-42e8-8948-2f83d1784d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" Aug 04 00:03:24.339360 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.339274 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" Aug 04 00:03:24.489377 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.489353 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2"] Aug 04 00:03:24.491767 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:03:24.491739 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de86e5b_857e_42e8_8948_2f83d1784d6d.slice/crio-f2307f10a5c020ecbe0b522f57e5147fbb645caea436a29f6ae7783411b968b3 WatchSource:0}: Error finding container f2307f10a5c020ecbe0b522f57e5147fbb645caea436a29f6ae7783411b968b3: Status 404 returned error can't find the container with id f2307f10a5c020ecbe0b522f57e5147fbb645caea436a29f6ae7783411b968b3 Aug 04 00:03:24.493750 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:24.493733 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:03:25.131670 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:25.131626 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" event={"ID":"3de86e5b-857e-42e8-8948-2f83d1784d6d","Type":"ContainerStarted","Data":"f2307f10a5c020ecbe0b522f57e5147fbb645caea436a29f6ae7783411b968b3"} Aug 04 00:03:27.144680 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:27.144655 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" event={"ID":"3de86e5b-857e-42e8-8948-2f83d1784d6d","Type":"ContainerStarted","Data":"b8561da347083b2d095612eba772f638ffa52ae4344702716375f546042a31f3"} Aug 04 00:03:27.158619 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:27.158569 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" podStartSLOduration=0.792476326 podStartE2EDuration="3.158551976s" totalImagesPullingTime="2.36607565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:27.157392722 +0000 UTC m=+2021.220649165" watchObservedRunningTime="2026-08-04 00:03:27.158551976 +0000 UTC m=+2021.221808405" Aug 04 00:03:28.151917 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:28.151881 2556 generic.go:356] "Generic (PLEG): container finished" podID="3de86e5b-857e-42e8-8948-2f83d1784d6d" containerID="b8561da347083b2d095612eba772f638ffa52ae4344702716375f546042a31f3" exitCode=0 Aug 04 00:03:28.152299 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:28.151973 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" event={"ID":"3de86e5b-857e-42e8-8948-2f83d1784d6d","Type":"ContainerDied","Data":"b8561da347083b2d095612eba772f638ffa52ae4344702716375f546042a31f3"} Aug 04 00:03:29.290688 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:29.290663 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" Aug 04 00:03:29.473239 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:29.473211 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de86e5b-857e-42e8-8948-2f83d1784d6d-maas-api-service-ca\") pod \"3de86e5b-857e-42e8-8948-2f83d1784d6d\" (UID: \"3de86e5b-857e-42e8-8948-2f83d1784d6d\") " Aug 04 00:03:29.473602 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:29.473580 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de86e5b-857e-42e8-8948-2f83d1784d6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3de86e5b-857e-42e8-8948-2f83d1784d6d" (UID: "3de86e5b-857e-42e8-8948-2f83d1784d6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:03:29.575007 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:29.574929 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de86e5b-857e-42e8-8948-2f83d1784d6d-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:03:30.164275 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:30.164244 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" Aug 04 00:03:30.164275 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:30.164260 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2" event={"ID":"3de86e5b-857e-42e8-8948-2f83d1784d6d","Type":"ContainerDied","Data":"f2307f10a5c020ecbe0b522f57e5147fbb645caea436a29f6ae7783411b968b3"} Aug 04 00:03:30.164499 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:30.164356 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2307f10a5c020ecbe0b522f57e5147fbb645caea436a29f6ae7783411b968b3" Aug 04 00:03:40.356593 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:40.356546 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2"] Aug 04 00:03:40.359449 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:40.359418 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9c1edcb-nddj2"] Aug 04 00:03:40.421301 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:03:40.421255 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de86e5b-857e-42e8-8948-2f83d1784d6d" path="/var/lib/kubelet/pods/3de86e5b-857e-42e8-8948-2f83d1784d6d/volumes" Aug 04 00:04:08.837134 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:08.837044 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86"] Aug 04 00:04:08.837613 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:08.837526 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de86e5b-857e-42e8-8948-2f83d1784d6d" containerName="revoke-keys" Aug 04 00:04:08.837613 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:08.837541 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de86e5b-857e-42e8-8948-2f83d1784d6d" containerName="revoke-keys" Aug 04 00:04:08.837697 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:08.837616 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de86e5b-857e-42e8-8948-2f83d1784d6d" containerName="revoke-keys" Aug 04 00:04:08.845069 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:08.845037 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" Aug 04 00:04:08.845387 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:08.845344 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86"] Aug 04 00:04:08.847784 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:08.847755 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:04:08.935929 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:08.935882 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3e3a-9f33-4b74-8752-4a94e4551d15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-50db8767-htw86\" (UID: \"2a5a3e3a-9f33-4b74-8752-4a94e4551d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" Aug 04 00:04:09.036613 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:09.036567 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3e3a-9f33-4b74-8752-4a94e4551d15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-50db8767-htw86\" (UID: \"2a5a3e3a-9f33-4b74-8752-4a94e4551d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" Aug 04 00:04:09.037262 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:09.037240 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3e3a-9f33-4b74-8752-4a94e4551d15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-50db8767-htw86\" (UID: \"2a5a3e3a-9f33-4b74-8752-4a94e4551d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" Aug 04 00:04:09.160182 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:09.160079 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" Aug 04 00:04:09.309424 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:09.309390 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86"] Aug 04 00:04:09.310717 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:04:09.310685 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5a3e3a_9f33_4b74_8752_4a94e4551d15.slice/crio-3fb48cf975781b9f16ab633c7832c15952b66325927fe04ecdc1931dcd7d0e09 WatchSource:0}: Error finding container 3fb48cf975781b9f16ab633c7832c15952b66325927fe04ecdc1931dcd7d0e09: Status 404 returned error can't find the container with id 3fb48cf975781b9f16ab633c7832c15952b66325927fe04ecdc1931dcd7d0e09 Aug 04 00:04:09.388834 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:09.388794 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" event={"ID":"2a5a3e3a-9f33-4b74-8752-4a94e4551d15","Type":"ContainerStarted","Data":"3fb48cf975781b9f16ab633c7832c15952b66325927fe04ecdc1931dcd7d0e09"} Aug 04 00:04:10.397171 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:10.397070 2556 generic.go:356] "Generic (PLEG): container finished" podID="2a5a3e3a-9f33-4b74-8752-4a94e4551d15" containerID="5ff11225b9b162fa5a36f1c3349ca572e782c44db904977b379803277f3f94b1" exitCode=0 Aug 04 00:04:10.397171 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:10.397146 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" event={"ID":"2a5a3e3a-9f33-4b74-8752-4a94e4551d15","Type":"ContainerDied","Data":"5ff11225b9b162fa5a36f1c3349ca572e782c44db904977b379803277f3f94b1"} Aug 04 00:04:11.551837 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:11.551805 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" Aug 04 00:04:11.660943 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:11.660833 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3e3a-9f33-4b74-8752-4a94e4551d15-maas-api-service-ca\") pod \"2a5a3e3a-9f33-4b74-8752-4a94e4551d15\" (UID: \"2a5a3e3a-9f33-4b74-8752-4a94e4551d15\") " Aug 04 00:04:11.661281 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:11.661258 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5a3e3a-9f33-4b74-8752-4a94e4551d15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a5a3e3a-9f33-4b74-8752-4a94e4551d15" (UID: "2a5a3e3a-9f33-4b74-8752-4a94e4551d15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:04:11.761592 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:11.761552 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3e3a-9f33-4b74-8752-4a94e4551d15-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:04:12.410287 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:12.410243 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" event={"ID":"2a5a3e3a-9f33-4b74-8752-4a94e4551d15","Type":"ContainerDied","Data":"3fb48cf975781b9f16ab633c7832c15952b66325927fe04ecdc1931dcd7d0e09"} Aug 04 00:04:12.410526 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:12.410339 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fb48cf975781b9f16ab633c7832c15952b66325927fe04ecdc1931dcd7d0e09" Aug 04 00:04:12.410526 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:12.410261 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86" Aug 04 00:04:14.671519 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:14.671474 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86"] Aug 04 00:04:14.673666 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:14.673641 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50db8767-htw86"] Aug 04 00:04:16.421173 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:16.421135 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5a3e3a-9f33-4b74-8752-4a94e4551d15" path="/var/lib/kubelet/pods/2a5a3e3a-9f33-4b74-8752-4a94e4551d15/volumes" Aug 04 00:04:43.700212 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.700179 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6"] Aug 04 00:04:43.700747 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.700555 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5a3e3a-9f33-4b74-8752-4a94e4551d15" containerName="revoke-keys" Aug 04 00:04:43.700747 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.700566 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5a3e3a-9f33-4b74-8752-4a94e4551d15" containerName="revoke-keys" Aug 04 00:04:43.700747 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.700632 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5a3e3a-9f33-4b74-8752-4a94e4551d15" containerName="revoke-keys" Aug 04 00:04:43.706162 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.706145 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" Aug 04 00:04:43.709087 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.708753 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:04:43.710650 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.710601 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6"] Aug 04 00:04:43.840425 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.840389 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f61d13-d281-4638-a587-ff746726a789-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6\" (UID: \"24f61d13-d281-4638-a587-ff746726a789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" Aug 04 00:04:43.941029 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.940997 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f61d13-d281-4638-a587-ff746726a789-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6\" (UID: \"24f61d13-d281-4638-a587-ff746726a789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" Aug 04 00:04:43.941634 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:43.941615 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f61d13-d281-4638-a587-ff746726a789-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6\" (UID: \"24f61d13-d281-4638-a587-ff746726a789\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" Aug 04 00:04:44.019046 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:44.019020 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" Aug 04 00:04:44.149978 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:44.149957 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6"] Aug 04 00:04:44.152160 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:04:44.152131 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f61d13_d281_4638_a587_ff746726a789.slice/crio-93540d7569ca49520ca2fe6ab14710329cba014fe2e850753903b637efcb3f4f WatchSource:0}: Error finding container 93540d7569ca49520ca2fe6ab14710329cba014fe2e850753903b637efcb3f4f: Status 404 returned error can't find the container with id 93540d7569ca49520ca2fe6ab14710329cba014fe2e850753903b637efcb3f4f Aug 04 00:04:44.597791 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:44.597698 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" event={"ID":"24f61d13-d281-4638-a587-ff746726a789","Type":"ContainerStarted","Data":"d2d7ba7418d4746fd00c321459533a50e1035fbe1782b1c4bdcb7294ed642537"} Aug 04 00:04:44.597791 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:44.597765 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" event={"ID":"24f61d13-d281-4638-a587-ff746726a789","Type":"ContainerStarted","Data":"93540d7569ca49520ca2fe6ab14710329cba014fe2e850753903b637efcb3f4f"} Aug 04 00:04:44.612424 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:44.612382 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" podStartSLOduration=1.612367156 podStartE2EDuration="1.612367156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:04:44.611173654 +0000 UTC m=+2098.674430075" watchObservedRunningTime="2026-08-04 00:04:44.612367156 +0000 UTC m=+2098.675623584" Aug 04 00:04:45.604397 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:45.604358 2556 generic.go:356] "Generic (PLEG): container finished" podID="24f61d13-d281-4638-a587-ff746726a789" containerID="d2d7ba7418d4746fd00c321459533a50e1035fbe1782b1c4bdcb7294ed642537" exitCode=0 Aug 04 00:04:45.604774 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:45.604439 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" event={"ID":"24f61d13-d281-4638-a587-ff746726a789","Type":"ContainerDied","Data":"d2d7ba7418d4746fd00c321459533a50e1035fbe1782b1c4bdcb7294ed642537"} Aug 04 00:04:46.742294 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:46.742268 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" Aug 04 00:04:46.753402 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:46.753378 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:04:46.758524 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:46.758502 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:04:46.868048 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:46.867974 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f61d13-d281-4638-a587-ff746726a789-maas-api-service-ca\") pod \"24f61d13-d281-4638-a587-ff746726a789\" (UID: \"24f61d13-d281-4638-a587-ff746726a789\") " Aug 04 00:04:46.868371 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:46.868350 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f61d13-d281-4638-a587-ff746726a789-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24f61d13-d281-4638-a587-ff746726a789" (UID: "24f61d13-d281-4638-a587-ff746726a789"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:04:46.969644 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:46.969620 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f61d13-d281-4638-a587-ff746726a789-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:04:47.616675 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:47.616641 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" Aug 04 00:04:47.616675 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:47.616664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6" event={"ID":"24f61d13-d281-4638-a587-ff746726a789","Type":"ContainerDied","Data":"93540d7569ca49520ca2fe6ab14710329cba014fe2e850753903b637efcb3f4f"} Aug 04 00:04:47.616879 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:47.616706 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93540d7569ca49520ca2fe6ab14710329cba014fe2e850753903b637efcb3f4f" Aug 04 00:04:49.677386 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:49.677352 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6"] Aug 04 00:04:49.682197 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:49.682160 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3113ce29-pf4z6"] Aug 04 00:04:50.420664 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:04:50.420631 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f61d13-d281-4638-a587-ff746726a789" path="/var/lib/kubelet/pods/24f61d13-d281-4638-a587-ff746726a789/volumes" Aug 04 00:05:24.146187 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.146141 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894"] Aug 04 00:05:24.146821 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.146784 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f61d13-d281-4638-a587-ff746726a789" containerName="revoke-keys" Aug 04 00:05:24.146821 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.146805 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f61d13-d281-4638-a587-ff746726a789" containerName="revoke-keys" Aug 04 00:05:24.146936 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.146900 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f61d13-d281-4638-a587-ff746726a789" containerName="revoke-keys" Aug 04 00:05:24.149793 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.149770 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" Aug 04 00:05:24.152092 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.152074 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:05:24.156931 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.156908 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894"] Aug 04 00:05:24.165870 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.165836 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e037e9-d754-451a-9d01-ec1a9fccf85e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894\" (UID: \"98e037e9-d754-451a-9d01-ec1a9fccf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" Aug 04 00:05:24.266326 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.266268 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e037e9-d754-451a-9d01-ec1a9fccf85e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894\" (UID: \"98e037e9-d754-451a-9d01-ec1a9fccf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" Aug 04 00:05:24.267012 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.266992 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e037e9-d754-451a-9d01-ec1a9fccf85e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894\" (UID: \"98e037e9-d754-451a-9d01-ec1a9fccf85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" Aug 04 00:05:24.463858 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.463832 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" Aug 04 00:05:24.602027 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.601997 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894"] Aug 04 00:05:24.603886 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:05:24.603844 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e037e9_d754_451a_9d01_ec1a9fccf85e.slice/crio-bcccf7a4a910e0840a2a0c39f5028afc7dc0c70b9af69b2027c78b099efcc476 WatchSource:0}: Error finding container bcccf7a4a910e0840a2a0c39f5028afc7dc0c70b9af69b2027c78b099efcc476: Status 404 returned error can't find the container with id bcccf7a4a910e0840a2a0c39f5028afc7dc0c70b9af69b2027c78b099efcc476 Aug 04 00:05:24.826462 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:24.826389 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" event={"ID":"98e037e9-d754-451a-9d01-ec1a9fccf85e","Type":"ContainerStarted","Data":"bcccf7a4a910e0840a2a0c39f5028afc7dc0c70b9af69b2027c78b099efcc476"} Aug 04 00:05:25.833612 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:25.833578 2556 generic.go:356] "Generic (PLEG): container finished" podID="98e037e9-d754-451a-9d01-ec1a9fccf85e" containerID="61439ae504828d8eae920c1839f5451990227ed06e1fa2b51c8f2175d5073ef5" exitCode=0 Aug 04 00:05:25.833988 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:25.833652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" event={"ID":"98e037e9-d754-451a-9d01-ec1a9fccf85e","Type":"ContainerDied","Data":"61439ae504828d8eae920c1839f5451990227ed06e1fa2b51c8f2175d5073ef5"} Aug 04 00:05:26.974154 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:26.974130 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" Aug 04 00:05:26.987574 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:26.987547 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e037e9-d754-451a-9d01-ec1a9fccf85e-maas-api-service-ca\") pod \"98e037e9-d754-451a-9d01-ec1a9fccf85e\" (UID: \"98e037e9-d754-451a-9d01-ec1a9fccf85e\") " Aug 04 00:05:26.987992 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:26.987959 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e037e9-d754-451a-9d01-ec1a9fccf85e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98e037e9-d754-451a-9d01-ec1a9fccf85e" (UID: "98e037e9-d754-451a-9d01-ec1a9fccf85e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:05:27.088385 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:27.088344 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e037e9-d754-451a-9d01-ec1a9fccf85e-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:05:27.846170 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:27.846141 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" Aug 04 00:05:27.846366 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:27.846173 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894" event={"ID":"98e037e9-d754-451a-9d01-ec1a9fccf85e","Type":"ContainerDied","Data":"bcccf7a4a910e0840a2a0c39f5028afc7dc0c70b9af69b2027c78b099efcc476"} Aug 04 00:05:27.846366 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:27.846223 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcccf7a4a910e0840a2a0c39f5028afc7dc0c70b9af69b2027c78b099efcc476" Aug 04 00:05:35.969739 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:35.969702 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894"] Aug 04 00:05:35.972048 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:35.972017 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b45e0cc-m6894"] Aug 04 00:05:36.421118 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:05:36.421040 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e037e9-d754-451a-9d01-ec1a9fccf85e" path="/var/lib/kubelet/pods/98e037e9-d754-451a-9d01-ec1a9fccf85e/volumes" Aug 04 00:08:08.889684 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:08.889605 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb"] Aug 04 00:08:08.890120 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:08.890086 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e037e9-d754-451a-9d01-ec1a9fccf85e" containerName="revoke-keys" Aug 04 00:08:08.890120 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:08.890099 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e037e9-d754-451a-9d01-ec1a9fccf85e" containerName="revoke-keys" Aug 04 00:08:08.890200 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:08.890168 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e037e9-d754-451a-9d01-ec1a9fccf85e" containerName="revoke-keys" Aug 04 00:08:08.893335 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:08.893296 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" Aug 04 00:08:08.895889 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:08.895871 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:08:08.901616 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:08.901592 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb"] Aug 04 00:08:08.937400 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:08.937360 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7dcdcca-b0d9-40da-b5b8-583b36615326-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb\" (UID: \"c7dcdcca-b0d9-40da-b5b8-583b36615326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" Aug 04 00:08:09.038154 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:09.038124 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7dcdcca-b0d9-40da-b5b8-583b36615326-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb\" (UID: \"c7dcdcca-b0d9-40da-b5b8-583b36615326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" Aug 04 00:08:09.038724 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:09.038707 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7dcdcca-b0d9-40da-b5b8-583b36615326-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb\" (UID: \"c7dcdcca-b0d9-40da-b5b8-583b36615326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" Aug 04 00:08:09.205022 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:09.204959 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" Aug 04 00:08:09.335642 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:09.335617 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb"] Aug 04 00:08:09.337767 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:08:09.337740 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7dcdcca_b0d9_40da_b5b8_583b36615326.slice/crio-494a7954264d8ec74825a5d5501b2e0782a95b6f64abee095dad6d89643cf6b5 WatchSource:0}: Error finding container 494a7954264d8ec74825a5d5501b2e0782a95b6f64abee095dad6d89643cf6b5: Status 404 returned error can't find the container with id 494a7954264d8ec74825a5d5501b2e0782a95b6f64abee095dad6d89643cf6b5 Aug 04 00:08:09.742682 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:09.742646 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" event={"ID":"c7dcdcca-b0d9-40da-b5b8-583b36615326","Type":"ContainerStarted","Data":"d6f646e0a6234d414ef5def3e1198953211aa4c77fb2bd54760ad2c98875bcb4"} Aug 04 00:08:09.742879 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:09.742699 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" event={"ID":"c7dcdcca-b0d9-40da-b5b8-583b36615326","Type":"ContainerStarted","Data":"494a7954264d8ec74825a5d5501b2e0782a95b6f64abee095dad6d89643cf6b5"} Aug 04 00:08:09.760578 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:09.760529 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" podStartSLOduration=1.76051664 podStartE2EDuration="1.76051664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:08:09.757961191 +0000 UTC m=+2303.821217622" watchObservedRunningTime="2026-08-04 00:08:09.76051664 +0000 UTC m=+2303.823773288" Aug 04 00:08:10.749274 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:10.749235 2556 generic.go:356] "Generic (PLEG): container finished" podID="c7dcdcca-b0d9-40da-b5b8-583b36615326" containerID="d6f646e0a6234d414ef5def3e1198953211aa4c77fb2bd54760ad2c98875bcb4" exitCode=0 Aug 04 00:08:10.749274 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:10.749275 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" event={"ID":"c7dcdcca-b0d9-40da-b5b8-583b36615326","Type":"ContainerDied","Data":"d6f646e0a6234d414ef5def3e1198953211aa4c77fb2bd54760ad2c98875bcb4"} Aug 04 00:08:11.887135 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:11.887112 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" Aug 04 00:08:11.965602 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:11.965561 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7dcdcca-b0d9-40da-b5b8-583b36615326-maas-api-service-ca\") pod \"c7dcdcca-b0d9-40da-b5b8-583b36615326\" (UID: \"c7dcdcca-b0d9-40da-b5b8-583b36615326\") " Aug 04 00:08:11.965889 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:11.965868 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7dcdcca-b0d9-40da-b5b8-583b36615326-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7dcdcca-b0d9-40da-b5b8-583b36615326" (UID: "c7dcdcca-b0d9-40da-b5b8-583b36615326"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:12.067134 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:12.067076 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7dcdcca-b0d9-40da-b5b8-583b36615326-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:08:12.761284 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:12.761262 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" Aug 04 00:08:12.761453 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:12.761301 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb" event={"ID":"c7dcdcca-b0d9-40da-b5b8-583b36615326","Type":"ContainerDied","Data":"494a7954264d8ec74825a5d5501b2e0782a95b6f64abee095dad6d89643cf6b5"} Aug 04 00:08:12.761453 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:12.761373 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="494a7954264d8ec74825a5d5501b2e0782a95b6f64abee095dad6d89643cf6b5" Aug 04 00:08:19.731277 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:19.731241 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb"] Aug 04 00:08:19.734907 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:19.734882 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c79f557-t8vrb"] Aug 04 00:08:20.421256 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:08:20.421219 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7dcdcca-b0d9-40da-b5b8-583b36615326" path="/var/lib/kubelet/pods/c7dcdcca-b0d9-40da-b5b8-583b36615326/volumes" Aug 04 00:09:46.585536 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:09:46.585507 2556 scope.go:117] "RemoveContainer" containerID="b8561da347083b2d095612eba772f638ffa52ae4344702716375f546042a31f3" Aug 04 00:09:46.792768 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:09:46.792678 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:09:46.801352 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:09:46.801300 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:10:46.607516 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:10:46.607480 2556 scope.go:117] "RemoveContainer" containerID="5ff11225b9b162fa5a36f1c3349ca572e782c44db904977b379803277f3f94b1" Aug 04 00:10:46.617963 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:10:46.617943 2556 scope.go:117] "RemoveContainer" containerID="d2d7ba7418d4746fd00c321459533a50e1035fbe1782b1c4bdcb7294ed642537" Aug 04 00:11:46.640078 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:46.640047 2556 scope.go:117] "RemoveContainer" containerID="61439ae504828d8eae920c1839f5451990227ed06e1fa2b51c8f2175d5073ef5" Aug 04 00:11:48.942588 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:48.942545 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn"] Aug 04 00:11:48.942951 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:48.942895 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7dcdcca-b0d9-40da-b5b8-583b36615326" containerName="revoke-keys" Aug 04 00:11:48.942951 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:48.942905 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7dcdcca-b0d9-40da-b5b8-583b36615326" containerName="revoke-keys" Aug 04 00:11:48.943022 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:48.942975 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7dcdcca-b0d9-40da-b5b8-583b36615326" containerName="revoke-keys" Aug 04 00:11:48.947216 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:48.947199 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" Aug 04 00:11:48.949488 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:48.949451 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:11:48.955823 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:48.955803 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn"] Aug 04 00:11:49.004652 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:49.004625 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6e37eb-e603-400e-bc6f-581389375ba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-335654eb-strrn\" (UID: \"da6e37eb-e603-400e-bc6f-581389375ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" Aug 04 00:11:49.105517 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:49.105486 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6e37eb-e603-400e-bc6f-581389375ba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-335654eb-strrn\" (UID: \"da6e37eb-e603-400e-bc6f-581389375ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" Aug 04 00:11:49.106055 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:49.106031 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6e37eb-e603-400e-bc6f-581389375ba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-335654eb-strrn\" (UID: \"da6e37eb-e603-400e-bc6f-581389375ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" Aug 04 00:11:49.258853 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:49.258833 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" Aug 04 00:11:49.410595 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:49.410569 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn"] Aug 04 00:11:49.412258 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:11:49.412231 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6e37eb_e603_400e_bc6f_581389375ba5.slice/crio-0dd438445dac5a3e2725c9f2da91dd5894fd6a2542b2eb0cbb38788054eeef23 WatchSource:0}: Error finding container 0dd438445dac5a3e2725c9f2da91dd5894fd6a2542b2eb0cbb38788054eeef23: Status 404 returned error can't find the container with id 0dd438445dac5a3e2725c9f2da91dd5894fd6a2542b2eb0cbb38788054eeef23 Aug 04 00:11:49.918965 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:49.918873 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" event={"ID":"da6e37eb-e603-400e-bc6f-581389375ba5","Type":"ContainerStarted","Data":"91d056ff138c36aa5b79210f339f6047ebaebeab1b0cc5732e0baf83024ad11f"} Aug 04 00:11:49.918965 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:49.918951 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" event={"ID":"da6e37eb-e603-400e-bc6f-581389375ba5","Type":"ContainerStarted","Data":"0dd438445dac5a3e2725c9f2da91dd5894fd6a2542b2eb0cbb38788054eeef23"} Aug 04 00:11:49.937397 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:49.937348 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" podStartSLOduration=1.937333893 podStartE2EDuration="1.937333893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:49.93590003 +0000 UTC m=+2523.999156475" watchObservedRunningTime="2026-08-04 00:11:49.937333893 +0000 UTC m=+2524.000590346" Aug 04 00:11:50.926129 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:50.926093 2556 generic.go:356] "Generic (PLEG): container finished" podID="da6e37eb-e603-400e-bc6f-581389375ba5" containerID="91d056ff138c36aa5b79210f339f6047ebaebeab1b0cc5732e0baf83024ad11f" exitCode=0 Aug 04 00:11:50.926725 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:50.926182 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" event={"ID":"da6e37eb-e603-400e-bc6f-581389375ba5","Type":"ContainerDied","Data":"91d056ff138c36aa5b79210f339f6047ebaebeab1b0cc5732e0baf83024ad11f"} Aug 04 00:11:52.068373 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:52.068352 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" Aug 04 00:11:52.127827 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:52.127800 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6e37eb-e603-400e-bc6f-581389375ba5-maas-api-service-ca\") pod \"da6e37eb-e603-400e-bc6f-581389375ba5\" (UID: \"da6e37eb-e603-400e-bc6f-581389375ba5\") " Aug 04 00:11:52.128120 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:52.128102 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6e37eb-e603-400e-bc6f-581389375ba5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da6e37eb-e603-400e-bc6f-581389375ba5" (UID: "da6e37eb-e603-400e-bc6f-581389375ba5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:11:52.228731 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:52.228705 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6e37eb-e603-400e-bc6f-581389375ba5-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:11:52.938969 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:52.938938 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" Aug 04 00:11:52.939161 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:52.938977 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn" event={"ID":"da6e37eb-e603-400e-bc6f-581389375ba5","Type":"ContainerDied","Data":"0dd438445dac5a3e2725c9f2da91dd5894fd6a2542b2eb0cbb38788054eeef23"} Aug 04 00:11:52.939161 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:11:52.939026 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd438445dac5a3e2725c9f2da91dd5894fd6a2542b2eb0cbb38788054eeef23" Aug 04 00:12:04.695471 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:12:04.695428 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn"] Aug 04 00:12:04.697654 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:12:04.697633 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-335654eb-strrn"] Aug 04 00:12:06.420554 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:12:06.420519 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6e37eb-e603-400e-bc6f-581389375ba5" path="/var/lib/kubelet/pods/da6e37eb-e603-400e-bc6f-581389375ba5/volumes" Aug 04 00:14:46.676179 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:14:46.676148 2556 scope.go:117] "RemoveContainer" containerID="d6f646e0a6234d414ef5def3e1198953211aa4c77fb2bd54760ad2c98875bcb4" Aug 04 00:14:46.826341 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:14:46.826251 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:14:46.835914 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:14:46.835896 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:15:00.132789 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.132760 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp"] Aug 04 00:15:00.133283 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.133181 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6e37eb-e603-400e-bc6f-581389375ba5" containerName="revoke-keys" Aug 04 00:15:00.133283 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.133199 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6e37eb-e603-400e-bc6f-581389375ba5" containerName="revoke-keys" Aug 04 00:15:00.133518 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.133327 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6e37eb-e603-400e-bc6f-581389375ba5" containerName="revoke-keys" Aug 04 00:15:00.136733 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.136713 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd"] Aug 04 00:15:00.136842 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.136824 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" Aug 04 00:15:00.139566 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.139544 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:15:00.140586 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.140567 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq"] Aug 04 00:15:00.140724 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.140705 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" Aug 04 00:15:00.144467 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.144446 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp"] Aug 04 00:15:00.144586 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.144531 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" Aug 04 00:15:00.147807 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.147371 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd"] Aug 04 00:15:00.151348 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.151149 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" Aug 04 00:15:00.159730 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.159710 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" Aug 04 00:15:00.160573 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.160550 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq"] Aug 04 00:15:00.196421 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.196333 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" Aug 04 00:15:00.314036 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.314008 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp"] Aug 04 00:15:00.317202 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:15:00.317154 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec2d0dd_720d_41a8_ae1d_e6c8936f6f97.slice/crio-ff811c051702c00b618ae86d89f96de3ba3b370995b54eb7e81e15d5cfa012a2 WatchSource:0}: Error finding container ff811c051702c00b618ae86d89f96de3ba3b370995b54eb7e81e15d5cfa012a2: Status 404 returned error can't find the container with id ff811c051702c00b618ae86d89f96de3ba3b370995b54eb7e81e15d5cfa012a2 Aug 04 00:15:00.365914 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.365886 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq"] Aug 04 00:15:00.367227 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:15:00.367202 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c55b58_f0cf_46c3_adee_73560ccca1b9.slice/crio-3ddab384574aa422974d5fc64143671f7569abe6ead53364c6142e2ddbcaab84 WatchSource:0}: Error finding container 3ddab384574aa422974d5fc64143671f7569abe6ead53364c6142e2ddbcaab84: Status 404 returned error can't find the container with id 3ddab384574aa422974d5fc64143671f7569abe6ead53364c6142e2ddbcaab84 Aug 04 00:15:00.547594 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.547570 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd"] Aug 04 00:15:00.548934 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:15:00.548909 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe0f254_05f0_4d0c_879d_eaed19ed1e7b.slice/crio-d2f37cb58f145b02f287be8bcf5a13ae6eab2b2caf696c6abe57eda06d1af51f WatchSource:0}: Error finding container d2f37cb58f145b02f287be8bcf5a13ae6eab2b2caf696c6abe57eda06d1af51f: Status 404 returned error can't find the container with id d2f37cb58f145b02f287be8bcf5a13ae6eab2b2caf696c6abe57eda06d1af51f Aug 04 00:15:00.933352 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.933319 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" event={"ID":"1fe0f254-05f0-4d0c-879d-eaed19ed1e7b","Type":"ContainerStarted","Data":"5f10d9d7894327dba3996afde23ba186ae35d0ac3b3a6ce2453cb74ec6d1767f"} Aug 04 00:15:00.933473 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.933382 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" event={"ID":"1fe0f254-05f0-4d0c-879d-eaed19ed1e7b","Type":"ContainerStarted","Data":"d2f37cb58f145b02f287be8bcf5a13ae6eab2b2caf696c6abe57eda06d1af51f"} Aug 04 00:15:00.934728 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.934707 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" event={"ID":"08c55b58-f0cf-46c3-adee-73560ccca1b9","Type":"ContainerStarted","Data":"d3db182b0e0510685e5cc265b738409bf158a05df0e03fb838597905206d5817"} Aug 04 00:15:00.934829 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.934746 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" event={"ID":"08c55b58-f0cf-46c3-adee-73560ccca1b9","Type":"ContainerStarted","Data":"3ddab384574aa422974d5fc64143671f7569abe6ead53364c6142e2ddbcaab84"} Aug 04 00:15:00.936169 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.936150 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" event={"ID":"cec2d0dd-720d-41a8-ae1d-e6c8936f6f97","Type":"ContainerStarted","Data":"a4949a6cb6020782e36e0cd23be3b0001e4754e1827ece7a297dce1285e31f74"} Aug 04 00:15:00.936253 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.936194 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" event={"ID":"cec2d0dd-720d-41a8-ae1d-e6c8936f6f97","Type":"ContainerStarted","Data":"ff811c051702c00b618ae86d89f96de3ba3b370995b54eb7e81e15d5cfa012a2"} Aug 04 00:15:00.947654 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.947615 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" podStartSLOduration=0.94760046 podStartE2EDuration="947.60046ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:00.946776758 +0000 UTC m=+2715.010033189" watchObservedRunningTime="2026-08-04 00:15:00.94760046 +0000 UTC m=+2715.010856889" Aug 04 00:15:00.961336 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.961282 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" podStartSLOduration=0.961270137 podStartE2EDuration="961.270137ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:00.960100167 +0000 UTC m=+2715.023356596" watchObservedRunningTime="2026-08-04 00:15:00.961270137 +0000 UTC m=+2715.024526634" Aug 04 00:15:00.973808 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:00.973768 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" podStartSLOduration=0.973755486 podStartE2EDuration="973.755486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:00.973372199 +0000 UTC m=+2715.036628627" watchObservedRunningTime="2026-08-04 00:15:00.973755486 +0000 UTC m=+2715.037011917" Aug 04 00:15:01.943094 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:01.943062 2556 generic.go:356] "Generic (PLEG): container finished" podID="08c55b58-f0cf-46c3-adee-73560ccca1b9" containerID="d3db182b0e0510685e5cc265b738409bf158a05df0e03fb838597905206d5817" exitCode=0 Aug 04 00:15:01.943443 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:01.943139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" event={"ID":"08c55b58-f0cf-46c3-adee-73560ccca1b9","Type":"ContainerDied","Data":"d3db182b0e0510685e5cc265b738409bf158a05df0e03fb838597905206d5817"} Aug 04 00:15:01.944607 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:01.944584 2556 generic.go:356] "Generic (PLEG): container finished" podID="cec2d0dd-720d-41a8-ae1d-e6c8936f6f97" containerID="a4949a6cb6020782e36e0cd23be3b0001e4754e1827ece7a297dce1285e31f74" exitCode=0 Aug 04 00:15:01.944724 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:01.944665 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" event={"ID":"cec2d0dd-720d-41a8-ae1d-e6c8936f6f97","Type":"ContainerDied","Data":"a4949a6cb6020782e36e0cd23be3b0001e4754e1827ece7a297dce1285e31f74"} Aug 04 00:15:01.946063 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:01.946044 2556 generic.go:356] "Generic (PLEG): container finished" podID="1fe0f254-05f0-4d0c-879d-eaed19ed1e7b" containerID="5f10d9d7894327dba3996afde23ba186ae35d0ac3b3a6ce2453cb74ec6d1767f" exitCode=0 Aug 04 00:15:01.946154 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:01.946089 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" event={"ID":"1fe0f254-05f0-4d0c-879d-eaed19ed1e7b","Type":"ContainerDied","Data":"5f10d9d7894327dba3996afde23ba186ae35d0ac3b3a6ce2453cb74ec6d1767f"} Aug 04 00:15:03.095448 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.095424 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" Aug 04 00:15:03.130875 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.130849 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" Aug 04 00:15:03.133980 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.133960 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" Aug 04 00:15:03.958238 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.958213 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" Aug 04 00:15:03.958425 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.958253 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd" event={"ID":"1fe0f254-05f0-4d0c-879d-eaed19ed1e7b","Type":"ContainerDied","Data":"d2f37cb58f145b02f287be8bcf5a13ae6eab2b2caf696c6abe57eda06d1af51f"} Aug 04 00:15:03.958425 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.958294 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2f37cb58f145b02f287be8bcf5a13ae6eab2b2caf696c6abe57eda06d1af51f" Aug 04 00:15:03.959641 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.959616 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" event={"ID":"08c55b58-f0cf-46c3-adee-73560ccca1b9","Type":"ContainerDied","Data":"3ddab384574aa422974d5fc64143671f7569abe6ead53364c6142e2ddbcaab84"} Aug 04 00:15:03.959770 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.959660 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ddab384574aa422974d5fc64143671f7569abe6ead53364c6142e2ddbcaab84" Aug 04 00:15:03.959770 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.959627 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq" Aug 04 00:15:03.960944 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.960927 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" Aug 04 00:15:03.961028 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.960949 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763375-blkwp" event={"ID":"cec2d0dd-720d-41a8-ae1d-e6c8936f6f97","Type":"ContainerDied","Data":"ff811c051702c00b618ae86d89f96de3ba3b370995b54eb7e81e15d5cfa012a2"} Aug 04 00:15:03.961028 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:15:03.960985 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff811c051702c00b618ae86d89f96de3ba3b370995b54eb7e81e15d5cfa012a2" Aug 04 00:16:09.589822 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.589746 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g"] Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590097 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe0f254-05f0-4d0c-879d-eaed19ed1e7b" containerName="cleanup" Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590108 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe0f254-05f0-4d0c-879d-eaed19ed1e7b" containerName="cleanup" Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590122 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec2d0dd-720d-41a8-ae1d-e6c8936f6f97" containerName="cleanup" Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590127 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec2d0dd-720d-41a8-ae1d-e6c8936f6f97" containerName="cleanup" Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590137 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c55b58-f0cf-46c3-adee-73560ccca1b9" containerName="cleanup" Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590142 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c55b58-f0cf-46c3-adee-73560ccca1b9" containerName="cleanup" Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590198 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec2d0dd-720d-41a8-ae1d-e6c8936f6f97" containerName="cleanup" Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590206 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c55b58-f0cf-46c3-adee-73560ccca1b9" containerName="cleanup" Aug 04 00:16:09.590231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.590214 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe0f254-05f0-4d0c-879d-eaed19ed1e7b" containerName="cleanup" Aug 04 00:16:09.593206 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.593190 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" Aug 04 00:16:09.596400 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.596183 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:16:09.602094 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.602068 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g"] Aug 04 00:16:09.694979 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.694950 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856fa6a4-5d42-4cb5-9753-99b9ea69a086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g\" (UID: \"856fa6a4-5d42-4cb5-9753-99b9ea69a086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" Aug 04 00:16:09.795953 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.795927 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856fa6a4-5d42-4cb5-9753-99b9ea69a086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g\" (UID: \"856fa6a4-5d42-4cb5-9753-99b9ea69a086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" Aug 04 00:16:09.796474 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.796457 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856fa6a4-5d42-4cb5-9753-99b9ea69a086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g\" (UID: \"856fa6a4-5d42-4cb5-9753-99b9ea69a086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" Aug 04 00:16:09.912842 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:09.912790 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" Aug 04 00:16:10.043683 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:10.043658 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g"] Aug 04 00:16:10.045955 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:16:10.045919 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856fa6a4_5d42_4cb5_9753_99b9ea69a086.slice/crio-e6d8902b8d6dba164827c572fc83649dc6dae028b7d8075e8e7d3939c966652f WatchSource:0}: Error finding container e6d8902b8d6dba164827c572fc83649dc6dae028b7d8075e8e7d3939c966652f: Status 404 returned error can't find the container with id e6d8902b8d6dba164827c572fc83649dc6dae028b7d8075e8e7d3939c966652f Aug 04 00:16:10.309814 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:10.309790 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" event={"ID":"856fa6a4-5d42-4cb5-9753-99b9ea69a086","Type":"ContainerStarted","Data":"e6d8902b8d6dba164827c572fc83649dc6dae028b7d8075e8e7d3939c966652f"} Aug 04 00:16:11.316529 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:11.316499 2556 generic.go:356] "Generic (PLEG): container finished" podID="856fa6a4-5d42-4cb5-9753-99b9ea69a086" containerID="a13b88d33dc3cd2f20a3aee1733708c9aa5b388a6c7bb3497d81011e65f581c0" exitCode=0 Aug 04 00:16:11.316903 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:11.316593 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" event={"ID":"856fa6a4-5d42-4cb5-9753-99b9ea69a086","Type":"ContainerDied","Data":"a13b88d33dc3cd2f20a3aee1733708c9aa5b388a6c7bb3497d81011e65f581c0"} Aug 04 00:16:12.446927 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:12.446902 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" Aug 04 00:16:12.519033 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:12.519007 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856fa6a4-5d42-4cb5-9753-99b9ea69a086-maas-api-service-ca\") pod \"856fa6a4-5d42-4cb5-9753-99b9ea69a086\" (UID: \"856fa6a4-5d42-4cb5-9753-99b9ea69a086\") " Aug 04 00:16:12.519355 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:12.519335 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856fa6a4-5d42-4cb5-9753-99b9ea69a086-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "856fa6a4-5d42-4cb5-9753-99b9ea69a086" (UID: "856fa6a4-5d42-4cb5-9753-99b9ea69a086"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:16:12.619814 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:12.619765 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856fa6a4-5d42-4cb5-9753-99b9ea69a086-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:16:13.327716 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:13.327691 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" Aug 04 00:16:13.327716 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:13.327704 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g" event={"ID":"856fa6a4-5d42-4cb5-9753-99b9ea69a086","Type":"ContainerDied","Data":"e6d8902b8d6dba164827c572fc83649dc6dae028b7d8075e8e7d3939c966652f"} Aug 04 00:16:13.327923 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:13.327745 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6d8902b8d6dba164827c572fc83649dc6dae028b7d8075e8e7d3939c966652f" Aug 04 00:16:13.356794 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:16:13.356768 2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856fa6a4_5d42_4cb5_9753_99b9ea69a086.slice\": RecentStats: unable to find data in memory cache]" Aug 04 00:16:20.036621 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:20.036588 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq"] Aug 04 00:16:20.042425 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:20.042402 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e43d99-29763375-rwgzq"] Aug 04 00:16:20.218628 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:20.218593 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g"] Aug 04 00:16:20.222184 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:20.222126 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e43d99-s5q5g"] Aug 04 00:16:20.420422 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:20.420356 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08c55b58-f0cf-46c3-adee-73560ccca1b9" path="/var/lib/kubelet/pods/08c55b58-f0cf-46c3-adee-73560ccca1b9/volumes" Aug 04 00:16:20.420638 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:20.420626 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856fa6a4-5d42-4cb5-9753-99b9ea69a086" path="/var/lib/kubelet/pods/856fa6a4-5d42-4cb5-9753-99b9ea69a086/volumes" Aug 04 00:16:27.215643 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.215611 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh"] Aug 04 00:16:27.216006 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.215953 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856fa6a4-5d42-4cb5-9753-99b9ea69a086" containerName="revoke-keys" Aug 04 00:16:27.216006 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.215964 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="856fa6a4-5d42-4cb5-9753-99b9ea69a086" containerName="revoke-keys" Aug 04 00:16:27.216081 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.216037 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="856fa6a4-5d42-4cb5-9753-99b9ea69a086" containerName="revoke-keys" Aug 04 00:16:27.218825 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.218809 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" Aug 04 00:16:27.221078 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.221057 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:16:27.225697 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.225665 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh"] Aug 04 00:16:27.330978 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.330946 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f93983-8ef3-4ead-a6ae-e6aacf495fcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh\" (UID: \"f8f93983-8ef3-4ead-a6ae-e6aacf495fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" Aug 04 00:16:27.431433 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.431407 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f93983-8ef3-4ead-a6ae-e6aacf495fcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh\" (UID: \"f8f93983-8ef3-4ead-a6ae-e6aacf495fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" Aug 04 00:16:27.431923 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.431906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f93983-8ef3-4ead-a6ae-e6aacf495fcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh\" (UID: \"f8f93983-8ef3-4ead-a6ae-e6aacf495fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" Aug 04 00:16:27.530608 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.530559 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" Aug 04 00:16:27.665067 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:27.665046 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh"] Aug 04 00:16:27.667003 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:16:27.666970 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f93983_8ef3_4ead_a6ae_e6aacf495fcb.slice/crio-cba6cc81c53fb11e42b7dd371d330fc861515a2637c6f50485f39ed72f385374 WatchSource:0}: Error finding container cba6cc81c53fb11e42b7dd371d330fc861515a2637c6f50485f39ed72f385374: Status 404 returned error can't find the container with id cba6cc81c53fb11e42b7dd371d330fc861515a2637c6f50485f39ed72f385374 Aug 04 00:16:28.413586 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:28.413555 2556 generic.go:356] "Generic (PLEG): container finished" podID="f8f93983-8ef3-4ead-a6ae-e6aacf495fcb" containerID="304c5e26d7a251d856521c0d72c7a39f89565fc154c3958a4ef8bea8bb1d04ff" exitCode=0 Aug 04 00:16:28.413931 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:28.413638 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" event={"ID":"f8f93983-8ef3-4ead-a6ae-e6aacf495fcb","Type":"ContainerDied","Data":"304c5e26d7a251d856521c0d72c7a39f89565fc154c3958a4ef8bea8bb1d04ff"} Aug 04 00:16:28.413931 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:28.413688 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" event={"ID":"f8f93983-8ef3-4ead-a6ae-e6aacf495fcb","Type":"ContainerStarted","Data":"cba6cc81c53fb11e42b7dd371d330fc861515a2637c6f50485f39ed72f385374"} Aug 04 00:16:29.540179 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:29.540157 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" Aug 04 00:16:29.648500 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:29.648474 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f93983-8ef3-4ead-a6ae-e6aacf495fcb-maas-api-service-ca\") pod \"f8f93983-8ef3-4ead-a6ae-e6aacf495fcb\" (UID: \"f8f93983-8ef3-4ead-a6ae-e6aacf495fcb\") " Aug 04 00:16:29.648825 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:29.648806 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f93983-8ef3-4ead-a6ae-e6aacf495fcb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8f93983-8ef3-4ead-a6ae-e6aacf495fcb" (UID: "f8f93983-8ef3-4ead-a6ae-e6aacf495fcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:16:29.749380 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:29.749356 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8f93983-8ef3-4ead-a6ae-e6aacf495fcb-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:16:30.425568 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:30.425540 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" Aug 04 00:16:30.425568 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:30.425559 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh" event={"ID":"f8f93983-8ef3-4ead-a6ae-e6aacf495fcb","Type":"ContainerDied","Data":"cba6cc81c53fb11e42b7dd371d330fc861515a2637c6f50485f39ed72f385374"} Aug 04 00:16:30.425749 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:30.425608 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba6cc81c53fb11e42b7dd371d330fc861515a2637c6f50485f39ed72f385374" Aug 04 00:16:37.916484 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:37.916454 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd"] Aug 04 00:16:37.920505 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:37.920478 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b0f5b6-29763375-qxffd"] Aug 04 00:16:38.093083 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:38.093046 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh"] Aug 04 00:16:38.096073 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:38.096046 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0f5b6-klvkh"] Aug 04 00:16:38.420662 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:38.420630 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe0f254-05f0-4d0c-879d-eaed19ed1e7b" path="/var/lib/kubelet/pods/1fe0f254-05f0-4d0c-879d-eaed19ed1e7b/volumes" Aug 04 00:16:38.420923 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:16:38.420909 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f93983-8ef3-4ead-a6ae-e6aacf495fcb" path="/var/lib/kubelet/pods/f8f93983-8ef3-4ead-a6ae-e6aacf495fcb/volumes" Aug 04 00:17:49.211389 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.211349 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg"] Aug 04 00:17:49.211923 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.211902 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f93983-8ef3-4ead-a6ae-e6aacf495fcb" containerName="revoke-keys" Aug 04 00:17:49.212006 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.211927 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f93983-8ef3-4ead-a6ae-e6aacf495fcb" containerName="revoke-keys" Aug 04 00:17:49.212059 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.212035 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f93983-8ef3-4ead-a6ae-e6aacf495fcb" containerName="revoke-keys" Aug 04 00:17:49.215175 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.215155 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" Aug 04 00:17:49.217619 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.217601 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:17:49.221930 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.221911 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg"] Aug 04 00:17:49.227542 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.227520 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d07e2e-a6e2-4b3c-b828-c74ef6601530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg\" (UID: \"03d07e2e-a6e2-4b3c-b828-c74ef6601530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" Aug 04 00:17:49.328530 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.328493 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d07e2e-a6e2-4b3c-b828-c74ef6601530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg\" (UID: \"03d07e2e-a6e2-4b3c-b828-c74ef6601530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" Aug 04 00:17:49.329095 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.329073 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d07e2e-a6e2-4b3c-b828-c74ef6601530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg\" (UID: \"03d07e2e-a6e2-4b3c-b828-c74ef6601530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" Aug 04 00:17:49.527141 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.527100 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" Aug 04 00:17:49.659375 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.659342 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg"] Aug 04 00:17:49.660812 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:17:49.660789 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d07e2e_a6e2_4b3c_b828_c74ef6601530.slice/crio-fb9904800850bdcf49171e7ef3b31c1516f4f2a6d4de49b254d6b410050446ad WatchSource:0}: Error finding container fb9904800850bdcf49171e7ef3b31c1516f4f2a6d4de49b254d6b410050446ad: Status 404 returned error can't find the container with id fb9904800850bdcf49171e7ef3b31c1516f4f2a6d4de49b254d6b410050446ad Aug 04 00:17:49.856024 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:49.855984 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" event={"ID":"03d07e2e-a6e2-4b3c-b828-c74ef6601530","Type":"ContainerStarted","Data":"fb9904800850bdcf49171e7ef3b31c1516f4f2a6d4de49b254d6b410050446ad"} Aug 04 00:17:50.863451 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:50.863412 2556 generic.go:356] "Generic (PLEG): container finished" podID="03d07e2e-a6e2-4b3c-b828-c74ef6601530" containerID="e69a4636e949efc6938ca0f549d02edf6fe9352e0dc4fbf14141749d79243825" exitCode=0 Aug 04 00:17:50.863849 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:50.863502 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" event={"ID":"03d07e2e-a6e2-4b3c-b828-c74ef6601530","Type":"ContainerDied","Data":"e69a4636e949efc6938ca0f549d02edf6fe9352e0dc4fbf14141749d79243825"} Aug 04 00:17:51.997914 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:51.997889 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" Aug 04 00:17:52.050431 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:52.050397 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d07e2e-a6e2-4b3c-b828-c74ef6601530-maas-api-service-ca\") pod \"03d07e2e-a6e2-4b3c-b828-c74ef6601530\" (UID: \"03d07e2e-a6e2-4b3c-b828-c74ef6601530\") " Aug 04 00:17:52.050797 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:52.050773 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d07e2e-a6e2-4b3c-b828-c74ef6601530-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03d07e2e-a6e2-4b3c-b828-c74ef6601530" (UID: "03d07e2e-a6e2-4b3c-b828-c74ef6601530"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:17:52.151661 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:52.151588 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d07e2e-a6e2-4b3c-b828-c74ef6601530-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:17:52.874721 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:52.874690 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" Aug 04 00:17:52.874721 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:52.874704 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg" event={"ID":"03d07e2e-a6e2-4b3c-b828-c74ef6601530","Type":"ContainerDied","Data":"fb9904800850bdcf49171e7ef3b31c1516f4f2a6d4de49b254d6b410050446ad"} Aug 04 00:17:52.874928 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:17:52.874746 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb9904800850bdcf49171e7ef3b31c1516f4f2a6d4de49b254d6b410050446ad" Aug 04 00:18:05.091430 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:05.091393 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg"] Aug 04 00:18:05.094516 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:05.094489 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b3817c-hwqtg"] Aug 04 00:18:06.420649 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:06.420616 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03d07e2e-a6e2-4b3c-b828-c74ef6601530" path="/var/lib/kubelet/pods/03d07e2e-a6e2-4b3c-b828-c74ef6601530/volumes" Aug 04 00:18:12.103457 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.103418 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr"] Aug 04 00:18:12.103853 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.103824 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d07e2e-a6e2-4b3c-b828-c74ef6601530" containerName="revoke-keys" Aug 04 00:18:12.103853 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.103835 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d07e2e-a6e2-4b3c-b828-c74ef6601530" containerName="revoke-keys" Aug 04 00:18:12.103923 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.103898 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d07e2e-a6e2-4b3c-b828-c74ef6601530" containerName="revoke-keys" Aug 04 00:18:12.107050 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.107034 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" Aug 04 00:18:12.109519 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.109499 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:18:12.114377 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.114347 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr"] Aug 04 00:18:12.224652 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.224612 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/410c0c65-1757-4718-8867-12e0a5affd53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-35e594-f4hwr\" (UID: \"410c0c65-1757-4718-8867-12e0a5affd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" Aug 04 00:18:12.325814 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.325784 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/410c0c65-1757-4718-8867-12e0a5affd53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-35e594-f4hwr\" (UID: \"410c0c65-1757-4718-8867-12e0a5affd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" Aug 04 00:18:12.326422 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.326402 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/410c0c65-1757-4718-8867-12e0a5affd53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-35e594-f4hwr\" (UID: \"410c0c65-1757-4718-8867-12e0a5affd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" Aug 04 00:18:12.418539 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.418431 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" Aug 04 00:18:12.554260 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.554234 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr"] Aug 04 00:18:12.556159 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:18:12.556132 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410c0c65_1757_4718_8867_12e0a5affd53.slice/crio-e7136a6089518b880f78ce963df4df45942dd753dcc2ce44a2c2827bc7ff42f5 WatchSource:0}: Error finding container e7136a6089518b880f78ce963df4df45942dd753dcc2ce44a2c2827bc7ff42f5: Status 404 returned error can't find the container with id e7136a6089518b880f78ce963df4df45942dd753dcc2ce44a2c2827bc7ff42f5 Aug 04 00:18:12.994541 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.994448 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" event={"ID":"410c0c65-1757-4718-8867-12e0a5affd53","Type":"ContainerStarted","Data":"969bbc79f4142672519e2014905570ca4689eda202af36017b67ce04884efc24"} Aug 04 00:18:12.994541 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:12.994515 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" event={"ID":"410c0c65-1757-4718-8867-12e0a5affd53","Type":"ContainerStarted","Data":"e7136a6089518b880f78ce963df4df45942dd753dcc2ce44a2c2827bc7ff42f5"} Aug 04 00:18:13.011544 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:13.011495 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" podStartSLOduration=1.011481121 podStartE2EDuration="1.011481121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:18:13.009225201 +0000 UTC m=+2907.072481632" watchObservedRunningTime="2026-08-04 00:18:13.011481121 +0000 UTC m=+2907.074737608" Aug 04 00:18:14.001552 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:14.001512 2556 generic.go:356] "Generic (PLEG): container finished" podID="410c0c65-1757-4718-8867-12e0a5affd53" containerID="969bbc79f4142672519e2014905570ca4689eda202af36017b67ce04884efc24" exitCode=0 Aug 04 00:18:14.002011 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:14.001605 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" event={"ID":"410c0c65-1757-4718-8867-12e0a5affd53","Type":"ContainerDied","Data":"969bbc79f4142672519e2014905570ca4689eda202af36017b67ce04884efc24"} Aug 04 00:18:15.138001 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:15.137977 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" Aug 04 00:18:15.251164 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:15.251129 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/410c0c65-1757-4718-8867-12e0a5affd53-maas-api-service-ca\") pod \"410c0c65-1757-4718-8867-12e0a5affd53\" (UID: \"410c0c65-1757-4718-8867-12e0a5affd53\") " Aug 04 00:18:15.251518 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:15.251497 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410c0c65-1757-4718-8867-12e0a5affd53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "410c0c65-1757-4718-8867-12e0a5affd53" (UID: "410c0c65-1757-4718-8867-12e0a5affd53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:18:15.352031 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:15.351946 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/410c0c65-1757-4718-8867-12e0a5affd53-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:18:16.016276 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:16.016248 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" Aug 04 00:18:16.016553 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:16.016297 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr" event={"ID":"410c0c65-1757-4718-8867-12e0a5affd53","Type":"ContainerDied","Data":"e7136a6089518b880f78ce963df4df45942dd753dcc2ce44a2c2827bc7ff42f5"} Aug 04 00:18:16.016553 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:16.016374 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7136a6089518b880f78ce963df4df45942dd753dcc2ce44a2c2827bc7ff42f5" Aug 04 00:18:27.855254 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:27.855214 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr"] Aug 04 00:18:27.857933 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:27.857907 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35e594-f4hwr"] Aug 04 00:18:28.421364 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:28.421332 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="410c0c65-1757-4718-8867-12e0a5affd53" path="/var/lib/kubelet/pods/410c0c65-1757-4718-8867-12e0a5affd53/volumes" Aug 04 00:18:46.724674 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:18:46.724644 2556 scope.go:117] "RemoveContainer" containerID="91d056ff138c36aa5b79210f339f6047ebaebeab1b0cc5732e0baf83024ad11f" Aug 04 00:19:46.867910 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:19:46.867803 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:19:46.877794 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:19:46.877766 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:21:46.764383 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:21:46.764345 2556 scope.go:117] "RemoveContainer" containerID="d3db182b0e0510685e5cc265b738409bf158a05df0e03fb838597905206d5817" Aug 04 00:21:46.773910 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:21:46.773887 2556 scope.go:117] "RemoveContainer" containerID="5f10d9d7894327dba3996afde23ba186ae35d0ac3b3a6ce2453cb74ec6d1767f" Aug 04 00:22:46.799524 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:22:46.799492 2556 scope.go:117] "RemoveContainer" containerID="304c5e26d7a251d856521c0d72c7a39f89565fc154c3958a4ef8bea8bb1d04ff" Aug 04 00:22:46.808778 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:22:46.808756 2556 scope.go:117] "RemoveContainer" containerID="a13b88d33dc3cd2f20a3aee1733708c9aa5b388a6c7bb3497d81011e65f581c0" Aug 04 00:24:46.848042 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:24:46.848009 2556 scope.go:117] "RemoveContainer" containerID="e69a4636e949efc6938ca0f549d02edf6fe9352e0dc4fbf14141749d79243825" Aug 04 00:24:46.857121 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:24:46.857100 2556 scope.go:117] "RemoveContainer" containerID="969bbc79f4142672519e2014905570ca4689eda202af36017b67ce04884efc24" Aug 04 00:24:46.910886 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:24:46.910841 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:24:46.924336 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:24:46.924297 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:27:32.545931 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.545896 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8"] Aug 04 00:27:32.546519 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.546500 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410c0c65-1757-4718-8867-12e0a5affd53" containerName="revoke-keys" Aug 04 00:27:32.546589 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.546522 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="410c0c65-1757-4718-8867-12e0a5affd53" containerName="revoke-keys" Aug 04 00:27:32.546643 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.546624 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="410c0c65-1757-4718-8867-12e0a5affd53" containerName="revoke-keys" Aug 04 00:27:32.550050 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.550031 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:32.552387 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.552355 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8c194a-serving-cert\"" Aug 04 00:27:32.552514 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.552422 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pzz5v\"" Aug 04 00:27:32.558169 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.558143 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8"] Aug 04 00:27:32.654520 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.654459 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:32.654520 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.654503 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzq98\" (UniqueName: \"kubernetes.io/projected/488ff5b5-332f-48ad-9962-b70e0760c27e-kube-api-access-zzq98\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:32.654739 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.654545 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/488ff5b5-332f-48ad-9962-b70e0760c27e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:32.755146 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.755112 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:32.755146 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.755147 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzq98\" (UniqueName: \"kubernetes.io/projected/488ff5b5-332f-48ad-9962-b70e0760c27e-kube-api-access-zzq98\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:32.755370 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.755183 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/488ff5b5-332f-48ad-9962-b70e0760c27e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:32.755370 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:27:32.755259 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-serving-cert: secret "maas-api-e2e-ipp-a-8c194a-serving-cert" not found Aug 04 00:27:32.755370 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:27:32.755344 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls podName:488ff5b5-332f-48ad-9962-b70e0760c27e nodeName:}" failed. No retries permitted until 2026-08-04 00:27:33.255324138 +0000 UTC m=+3467.318580567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls") pod "maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" (UID: "488ff5b5-332f-48ad-9962-b70e0760c27e") : secret "maas-api-e2e-ipp-a-8c194a-serving-cert" not found Aug 04 00:27:32.755875 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.755858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/488ff5b5-332f-48ad-9962-b70e0760c27e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:32.763641 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:32.763610 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzq98\" (UniqueName: \"kubernetes.io/projected/488ff5b5-332f-48ad-9962-b70e0760c27e-kube-api-access-zzq98\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:33.163047 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.163009 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h"] Aug 04 00:27:33.168520 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.168488 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:33.171619 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.171597 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8c194a\"" Aug 04 00:27:33.177024 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.177002 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h"] Aug 04 00:27:33.214797 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.214765 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt"] Aug 04 00:27:33.218088 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.218068 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:33.220776 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.220757 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8c194a-dockercfg-2tmv4\"" Aug 04 00:27:33.227967 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.227945 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt"] Aug 04 00:27:33.260285 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.260262 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:33.260410 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.260364 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f8l\" (UniqueName: \"kubernetes.io/projected/aab3cb40-4eae-4035-a944-006dd9549a30-kube-api-access-58f8l\") pod \"payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h\" (UID: \"aab3cb40-4eae-4035-a944-006dd9549a30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:33.260410 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.260395 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab3cb40-4eae-4035-a944-006dd9549a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h\" (UID: \"aab3cb40-4eae-4035-a944-006dd9549a30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:33.262731 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.262710 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:33.361532 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.361497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f8l\" (UniqueName: \"kubernetes.io/projected/aab3cb40-4eae-4035-a944-006dd9549a30-kube-api-access-58f8l\") pod \"payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h\" (UID: \"aab3cb40-4eae-4035-a944-006dd9549a30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:33.361691 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.361539 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt\" (UID: \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:33.361691 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.361563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab3cb40-4eae-4035-a944-006dd9549a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h\" (UID: \"aab3cb40-4eae-4035-a944-006dd9549a30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:33.361691 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.361585 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9dw\" (UniqueName: \"kubernetes.io/projected/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-kube-api-access-4f9dw\") pod \"payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt\" (UID: \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:33.362196 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.362177 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab3cb40-4eae-4035-a944-006dd9549a30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h\" (UID: \"aab3cb40-4eae-4035-a944-006dd9549a30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:33.370422 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.370396 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f8l\" (UniqueName: \"kubernetes.io/projected/aab3cb40-4eae-4035-a944-006dd9549a30-kube-api-access-58f8l\") pod \"payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h\" (UID: \"aab3cb40-4eae-4035-a944-006dd9549a30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:33.462592 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.462565 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt\" (UID: \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:33.462718 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.462612 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9dw\" (UniqueName: \"kubernetes.io/projected/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-kube-api-access-4f9dw\") pod \"payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt\" (UID: \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:33.463292 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.463273 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt\" (UID: \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:33.464517 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.464497 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:33.470526 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.470502 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9dw\" (UniqueName: \"kubernetes.io/projected/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-kube-api-access-4f9dw\") pod \"payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt\" (UID: \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:33.482370 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.482347 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:33.529405 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.529370 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:33.630504 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.630471 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8"] Aug 04 00:27:33.633510 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:27:33.633481 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488ff5b5_332f_48ad_9962_b70e0760c27e.slice/crio-039c17ad581271755749a7823d68ac0baa3849cf923b2101030aad92365f7403 WatchSource:0}: Error finding container 039c17ad581271755749a7823d68ac0baa3849cf923b2101030aad92365f7403: Status 404 returned error can't find the container with id 039c17ad581271755749a7823d68ac0baa3849cf923b2101030aad92365f7403 Aug 04 00:27:33.634259 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.634234 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:27:33.660191 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.660160 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h"] Aug 04 00:27:33.660822 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:27:33.660787 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab3cb40_4eae_4035_a944_006dd9549a30.slice/crio-22c0c0950584432d6c90cf7522847678381f5422e3674a58f2d2de099bcf1d54 WatchSource:0}: Error finding container 22c0c0950584432d6c90cf7522847678381f5422e3674a58f2d2de099bcf1d54: Status 404 returned error can't find the container with id 22c0c0950584432d6c90cf7522847678381f5422e3674a58f2d2de099bcf1d54 Aug 04 00:27:33.696025 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:33.696002 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt"] Aug 04 00:27:33.697930 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:27:33.697904 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cb1a68_b1a4_45ce_9cbe_1fd9f34e11ef.slice/crio-5c0500532284e35a84f2f48a8e134c61c9d7e33406a4a6912fc2f00e10c78179 WatchSource:0}: Error finding container 5c0500532284e35a84f2f48a8e134c61c9d7e33406a4a6912fc2f00e10c78179: Status 404 returned error can't find the container with id 5c0500532284e35a84f2f48a8e134c61c9d7e33406a4a6912fc2f00e10c78179 Aug 04 00:27:34.109662 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:34.109623 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" event={"ID":"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef","Type":"ContainerStarted","Data":"5c0500532284e35a84f2f48a8e134c61c9d7e33406a4a6912fc2f00e10c78179"} Aug 04 00:27:34.110893 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:34.110860 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" event={"ID":"488ff5b5-332f-48ad-9962-b70e0760c27e","Type":"ContainerStarted","Data":"039c17ad581271755749a7823d68ac0baa3849cf923b2101030aad92365f7403"} Aug 04 00:27:34.112057 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:34.112033 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" event={"ID":"aab3cb40-4eae-4035-a944-006dd9549a30","Type":"ContainerStarted","Data":"22c0c0950584432d6c90cf7522847678381f5422e3674a58f2d2de099bcf1d54"} Aug 04 00:27:37.134717 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.134615 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" event={"ID":"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef","Type":"ContainerStarted","Data":"300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc"} Aug 04 00:27:37.135129 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.134741 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:37.136278 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.136253 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" event={"ID":"488ff5b5-332f-48ad-9962-b70e0760c27e","Type":"ContainerStarted","Data":"1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377"} Aug 04 00:27:37.136427 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.136331 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:37.137982 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.137958 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" event={"ID":"aab3cb40-4eae-4035-a944-006dd9549a30","Type":"ContainerStarted","Data":"76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863"} Aug 04 00:27:37.138088 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.138007 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:27:37.152951 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.152901 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" podStartSLOduration=1.07144557 podStartE2EDuration="4.152883534s" totalImagesPullingTime="3.081437964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:27:37.151132563 +0000 UTC m=+3471.214388996" watchObservedRunningTime="2026-08-04 00:27:37.152883534 +0000 UTC m=+3471.216139970" Aug 04 00:27:37.173579 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.173531 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" podStartSLOduration=2.026590444 podStartE2EDuration="5.173517643s" totalImagesPullingTime="3.146927199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:27:37.173156358 +0000 UTC m=+3471.236412788" watchObservedRunningTime="2026-08-04 00:27:37.173517643 +0000 UTC m=+3471.236774073" Aug 04 00:27:37.187710 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:37.187668 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" podStartSLOduration=1.062274604 podStartE2EDuration="4.187655181s" totalImagesPullingTime="3.125380577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:27:37.184999252 +0000 UTC m=+3471.248255683" watchObservedRunningTime="2026-08-04 00:27:37.187655181 +0000 UTC m=+3471.250911618" Aug 04 00:27:43.151123 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:43.151096 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:27:48.147193 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:48.147161 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:27:48.147684 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:27:48.147285 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:28:01.792080 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:01.792044 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl"] Aug 04 00:28:01.795247 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:01.795221 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:01.797622 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:01.797594 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f96979-serving-cert\"" Aug 04 00:28:01.805081 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:01.805053 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl"] Aug 04 00:28:01.904401 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:01.904362 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36d0f13a-0452-48d0-803e-fa728e67e41e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:01.904572 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:01.904422 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d0f13a-0452-48d0-803e-fa728e67e41e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:01.904572 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:01.904515 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cs72\" (UniqueName: \"kubernetes.io/projected/36d0f13a-0452-48d0-803e-fa728e67e41e-kube-api-access-6cs72\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:02.005504 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.005470 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36d0f13a-0452-48d0-803e-fa728e67e41e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:02.005504 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.005510 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d0f13a-0452-48d0-803e-fa728e67e41e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:02.005775 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.005558 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cs72\" (UniqueName: \"kubernetes.io/projected/36d0f13a-0452-48d0-803e-fa728e67e41e-kube-api-access-6cs72\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:02.006282 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.006252 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d0f13a-0452-48d0-803e-fa728e67e41e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:02.007985 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.007963 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36d0f13a-0452-48d0-803e-fa728e67e41e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:02.014092 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.014065 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cs72\" (UniqueName: \"kubernetes.io/projected/36d0f13a-0452-48d0-803e-fa728e67e41e-kube-api-access-6cs72\") pod \"maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:02.108995 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.108905 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:02.248139 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.248106 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl"] Aug 04 00:28:02.249356 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:28:02.249324 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d0f13a_0452_48d0_803e_fa728e67e41e.slice/crio-19ad29e48cae7f0dcaa292d73f6934e7f0a5422aeb72b24d1ba1a26282b33bdc WatchSource:0}: Error finding container 19ad29e48cae7f0dcaa292d73f6934e7f0a5422aeb72b24d1ba1a26282b33bdc: Status 404 returned error can't find the container with id 19ad29e48cae7f0dcaa292d73f6934e7f0a5422aeb72b24d1ba1a26282b33bdc Aug 04 00:28:02.279351 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.279294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" event={"ID":"36d0f13a-0452-48d0-803e-fa728e67e41e","Type":"ContainerStarted","Data":"19ad29e48cae7f0dcaa292d73f6934e7f0a5422aeb72b24d1ba1a26282b33bdc"} Aug 04 00:28:02.435757 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.435668 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4"] Aug 04 00:28:02.440130 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.440107 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:02.442448 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.442427 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f96979\"" Aug 04 00:28:02.450472 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.450445 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4"] Aug 04 00:28:02.493062 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.493026 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr"] Aug 04 00:28:02.496680 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.496660 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:02.498986 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.498962 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f96979-dockercfg-ch6d8\"" Aug 04 00:28:02.505140 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.505106 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr"] Aug 04 00:28:02.511670 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.511646 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594e8702-5242-4454-aa12-05fd73eb0c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4\" (UID: \"594e8702-5242-4454-aa12-05fd73eb0c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:02.511800 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.511692 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7z5\" (UniqueName: \"kubernetes.io/projected/594e8702-5242-4454-aa12-05fd73eb0c4e-kube-api-access-kx7z5\") pod \"payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4\" (UID: \"594e8702-5242-4454-aa12-05fd73eb0c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:02.612458 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.612420 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594e8702-5242-4454-aa12-05fd73eb0c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4\" (UID: \"594e8702-5242-4454-aa12-05fd73eb0c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:02.612662 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.612482 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11de2616-b136-432b-a77e-ebf6fee94fb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr\" (UID: \"11de2616-b136-432b-a77e-ebf6fee94fb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:02.612662 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.612516 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7z5\" (UniqueName: \"kubernetes.io/projected/594e8702-5242-4454-aa12-05fd73eb0c4e-kube-api-access-kx7z5\") pod \"payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4\" (UID: \"594e8702-5242-4454-aa12-05fd73eb0c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:02.612783 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.612660 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gms9\" (UniqueName: \"kubernetes.io/projected/11de2616-b136-432b-a77e-ebf6fee94fb3-kube-api-access-9gms9\") pod \"payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr\" (UID: \"11de2616-b136-432b-a77e-ebf6fee94fb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:02.613203 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.613185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594e8702-5242-4454-aa12-05fd73eb0c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4\" (UID: \"594e8702-5242-4454-aa12-05fd73eb0c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:02.621089 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.621066 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7z5\" (UniqueName: \"kubernetes.io/projected/594e8702-5242-4454-aa12-05fd73eb0c4e-kube-api-access-kx7z5\") pod \"payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4\" (UID: \"594e8702-5242-4454-aa12-05fd73eb0c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:02.713530 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.713502 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gms9\" (UniqueName: \"kubernetes.io/projected/11de2616-b136-432b-a77e-ebf6fee94fb3-kube-api-access-9gms9\") pod \"payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr\" (UID: \"11de2616-b136-432b-a77e-ebf6fee94fb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:02.713703 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.713564 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11de2616-b136-432b-a77e-ebf6fee94fb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr\" (UID: \"11de2616-b136-432b-a77e-ebf6fee94fb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:02.714240 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.714221 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11de2616-b136-432b-a77e-ebf6fee94fb3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr\" (UID: \"11de2616-b136-432b-a77e-ebf6fee94fb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:02.721594 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.721572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gms9\" (UniqueName: \"kubernetes.io/projected/11de2616-b136-432b-a77e-ebf6fee94fb3-kube-api-access-9gms9\") pod \"payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr\" (UID: \"11de2616-b136-432b-a77e-ebf6fee94fb3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:02.753569 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.753535 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:02.809449 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.809418 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:02.901924 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.901897 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4"] Aug 04 00:28:02.905029 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:28:02.904993 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594e8702_5242_4454_aa12_05fd73eb0c4e.slice/crio-a4de6d5c21558c784df2761f9d9da0581214e8953a91fcf6d6218225b4019529 WatchSource:0}: Error finding container a4de6d5c21558c784df2761f9d9da0581214e8953a91fcf6d6218225b4019529: Status 404 returned error can't find the container with id a4de6d5c21558c784df2761f9d9da0581214e8953a91fcf6d6218225b4019529 Aug 04 00:28:02.970003 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:02.969973 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr"] Aug 04 00:28:02.971833 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:28:02.971807 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11de2616_b136_432b_a77e_ebf6fee94fb3.slice/crio-b21bd2e50cbed92e8a107376536549fc103dfb781afd8610e4235e926b9ae599 WatchSource:0}: Error finding container b21bd2e50cbed92e8a107376536549fc103dfb781afd8610e4235e926b9ae599: Status 404 returned error can't find the container with id b21bd2e50cbed92e8a107376536549fc103dfb781afd8610e4235e926b9ae599 Aug 04 00:28:03.287122 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.287010 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" event={"ID":"11de2616-b136-432b-a77e-ebf6fee94fb3","Type":"ContainerStarted","Data":"cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be"} Aug 04 00:28:03.287122 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.287062 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" event={"ID":"11de2616-b136-432b-a77e-ebf6fee94fb3","Type":"ContainerStarted","Data":"b21bd2e50cbed92e8a107376536549fc103dfb781afd8610e4235e926b9ae599"} Aug 04 00:28:03.287382 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.287136 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:03.288563 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.288536 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" event={"ID":"36d0f13a-0452-48d0-803e-fa728e67e41e","Type":"ContainerStarted","Data":"1b3f12b07f114f9336098e75c6ff1d43beaa9ee7a0d7be3947711c57ca81bd88"} Aug 04 00:28:03.288703 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.288601 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:03.290240 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.290215 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" event={"ID":"594e8702-5242-4454-aa12-05fd73eb0c4e","Type":"ContainerStarted","Data":"e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea"} Aug 04 00:28:03.290396 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.290263 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" event={"ID":"594e8702-5242-4454-aa12-05fd73eb0c4e","Type":"ContainerStarted","Data":"a4de6d5c21558c784df2761f9d9da0581214e8953a91fcf6d6218225b4019529"} Aug 04 00:28:03.290396 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.290296 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:03.304237 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.304183 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" podStartSLOduration=1.304165792 podStartE2EDuration="1.304165792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:28:03.302702006 +0000 UTC m=+3497.365958448" watchObservedRunningTime="2026-08-04 00:28:03.304165792 +0000 UTC m=+3497.367422222" Aug 04 00:28:03.319259 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.319192 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" podStartSLOduration=1.7904825199999999 podStartE2EDuration="2.319172005s" totalImagesPullingTime="528.689485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:28:03.318368388 +0000 UTC m=+3497.381624819" watchObservedRunningTime="2026-08-04 00:28:03.319172005 +0000 UTC m=+3497.382428435" Aug 04 00:28:03.333568 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:03.333518 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" podStartSLOduration=1.333499006 podStartE2EDuration="1.333499006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:28:03.332595983 +0000 UTC m=+3497.395852417" watchObservedRunningTime="2026-08-04 00:28:03.333499006 +0000 UTC m=+3497.396755436" Aug 04 00:28:09.301642 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:09.301563 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:28:14.297081 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:14.297051 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:28:14.297627 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:14.297357 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:28:32.507521 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:28:32.507466 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr_11de2616-b136-432b-a77e-ebf6fee94fb3/payload-processing/0.log" Aug 04 00:29:15.070278 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:15.070237 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt_b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef/payload-processing/0.log" Aug 04 00:29:15.306521 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:15.306493 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr_11de2616-b136-432b-a77e-ebf6fee94fb3/payload-processing/0.log" Aug 04 00:29:20.278036 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.278003 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz"] Aug 04 00:29:20.280638 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.280619 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:20.282962 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.282938 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-a51f42-serving-cert\"" Aug 04 00:29:20.292832 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.292813 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz"] Aug 04 00:29:20.408151 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.408116 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrz8t\" (UniqueName: \"kubernetes.io/projected/c8072820-a124-41d5-8041-3f1ea855cad1-kube-api-access-zrz8t\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:20.408332 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.408164 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8072820-a124-41d5-8041-3f1ea855cad1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:20.408332 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.408205 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:20.509459 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.509417 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:20.509656 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:29:20.509564 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-serving-cert: secret "maas-api-e2e-ipp-cleanup-a51f42-serving-cert" not found Aug 04 00:29:20.509656 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.509564 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrz8t\" (UniqueName: \"kubernetes.io/projected/c8072820-a124-41d5-8041-3f1ea855cad1-kube-api-access-zrz8t\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:20.509656 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:29:20.509631 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls podName:c8072820-a124-41d5-8041-3f1ea855cad1 nodeName:}" failed. No retries permitted until 2026-08-04 00:29:21.009612406 +0000 UTC m=+3575.072868814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" (UID: "c8072820-a124-41d5-8041-3f1ea855cad1") : secret "maas-api-e2e-ipp-cleanup-a51f42-serving-cert" not found Aug 04 00:29:20.509841 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.509676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8072820-a124-41d5-8041-3f1ea855cad1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:20.510398 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.510369 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8072820-a124-41d5-8041-3f1ea855cad1-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:20.521888 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:20.521858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrz8t\" (UniqueName: \"kubernetes.io/projected/c8072820-a124-41d5-8041-3f1ea855cad1-kube-api-access-zrz8t\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:21.014612 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:21.014576 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:21.017270 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:21.017238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:21.194209 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:21.194161 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:21.328943 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:21.328904 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz"] Aug 04 00:29:21.330435 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:29:21.330407 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8072820_a124_41d5_8041_3f1ea855cad1.slice/crio-94876a0d8909c89fd6dae39ee87f47f3889dd4741e1eb840d81e92b434ee858f WatchSource:0}: Error finding container 94876a0d8909c89fd6dae39ee87f47f3889dd4741e1eb840d81e92b434ee858f: Status 404 returned error can't find the container with id 94876a0d8909c89fd6dae39ee87f47f3889dd4741e1eb840d81e92b434ee858f Aug 04 00:29:21.731890 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:21.731852 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" event={"ID":"c8072820-a124-41d5-8041-3f1ea855cad1","Type":"ContainerStarted","Data":"94876a0d8909c89fd6dae39ee87f47f3889dd4741e1eb840d81e92b434ee858f"} Aug 04 00:29:22.738750 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:22.738713 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" event={"ID":"c8072820-a124-41d5-8041-3f1ea855cad1","Type":"ContainerStarted","Data":"ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25"} Aug 04 00:29:22.739245 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:22.738870 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:22.754665 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:22.754610 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" podStartSLOduration=2.268295309 podStartE2EDuration="2.754589674s" totalImagesPullingTime="486.294365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:29:22.754376853 +0000 UTC m=+3576.817633302" watchObservedRunningTime="2026-08-04 00:29:22.754589674 +0000 UTC m=+3576.817846105" Aug 04 00:29:28.750693 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:28.750661 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:44.666243 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:44.666203 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth"] Aug 04 00:29:44.669073 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:44.669056 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" Aug 04 00:29:44.671264 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:44.671246 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:29:44.677345 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:44.677278 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth"] Aug 04 00:29:44.731925 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:44.731894 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e919f839-4c21-412b-a23f-ce80fd2f76ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth\" (UID: \"e919f839-4c21-412b-a23f-ce80fd2f76ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" Aug 04 00:29:44.832646 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:44.832600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e919f839-4c21-412b-a23f-ce80fd2f76ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth\" (UID: \"e919f839-4c21-412b-a23f-ce80fd2f76ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" Aug 04 00:29:44.833247 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:44.833224 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e919f839-4c21-412b-a23f-ce80fd2f76ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth\" (UID: \"e919f839-4c21-412b-a23f-ce80fd2f76ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" Aug 04 00:29:44.982800 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:44.982768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" Aug 04 00:29:45.118455 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:45.118421 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth"] Aug 04 00:29:45.120557 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:29:45.120527 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode919f839_4c21_412b_a23f_ce80fd2f76ac.slice/crio-b9a20ab8e3b5c5f48c8f675f58845ccd3be738a04c47b51292e0210f4f010357 WatchSource:0}: Error finding container b9a20ab8e3b5c5f48c8f675f58845ccd3be738a04c47b51292e0210f4f010357: Status 404 returned error can't find the container with id b9a20ab8e3b5c5f48c8f675f58845ccd3be738a04c47b51292e0210f4f010357 Aug 04 00:29:45.871911 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:45.871878 2556 generic.go:356] "Generic (PLEG): container finished" podID="e919f839-4c21-412b-a23f-ce80fd2f76ac" containerID="fd1e0022c1711140ac218479bcc86f09ccb9905a7697ad1e4f1d5955ce87f3bb" exitCode=0 Aug 04 00:29:45.872367 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:45.871965 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" event={"ID":"e919f839-4c21-412b-a23f-ce80fd2f76ac","Type":"ContainerDied","Data":"fd1e0022c1711140ac218479bcc86f09ccb9905a7697ad1e4f1d5955ce87f3bb"} Aug 04 00:29:45.872367 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:45.872030 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" event={"ID":"e919f839-4c21-412b-a23f-ce80fd2f76ac","Type":"ContainerStarted","Data":"b9a20ab8e3b5c5f48c8f675f58845ccd3be738a04c47b51292e0210f4f010357"} Aug 04 00:29:46.970229 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:46.970203 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:29:46.984003 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:46.983972 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:29:47.025115 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:47.025093 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" Aug 04 00:29:47.156334 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:47.156224 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e919f839-4c21-412b-a23f-ce80fd2f76ac-maas-api-service-ca\") pod \"e919f839-4c21-412b-a23f-ce80fd2f76ac\" (UID: \"e919f839-4c21-412b-a23f-ce80fd2f76ac\") " Aug 04 00:29:47.156582 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:47.156556 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e919f839-4c21-412b-a23f-ce80fd2f76ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e919f839-4c21-412b-a23f-ce80fd2f76ac" (UID: "e919f839-4c21-412b-a23f-ce80fd2f76ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:29:47.257039 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:47.257000 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e919f839-4c21-412b-a23f-ce80fd2f76ac-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:29:47.884296 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:47.884262 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" Aug 04 00:29:47.884501 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:47.884260 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth" event={"ID":"e919f839-4c21-412b-a23f-ce80fd2f76ac","Type":"ContainerDied","Data":"b9a20ab8e3b5c5f48c8f675f58845ccd3be738a04c47b51292e0210f4f010357"} Aug 04 00:29:47.884501 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:47.884374 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9a20ab8e3b5c5f48c8f675f58845ccd3be738a04c47b51292e0210f4f010357" Aug 04 00:29:50.638424 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.638384 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz"] Aug 04 00:29:50.638855 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.638663 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" podUID="c8072820-a124-41d5-8041-3f1ea855cad1" containerName="maas-api" containerID="cri-o://ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25" gracePeriod=30 Aug 04 00:29:50.841008 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.840968 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth"] Aug 04 00:29:50.843933 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.843901 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a51f42-zgxth"] Aug 04 00:29:50.902072 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.901991 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:50.902372 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.902351 2556 generic.go:356] "Generic (PLEG): container finished" podID="c8072820-a124-41d5-8041-3f1ea855cad1" containerID="ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25" exitCode=0 Aug 04 00:29:50.902452 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.902385 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" event={"ID":"c8072820-a124-41d5-8041-3f1ea855cad1","Type":"ContainerDied","Data":"ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25"} Aug 04 00:29:50.902452 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.902425 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" event={"ID":"c8072820-a124-41d5-8041-3f1ea855cad1","Type":"ContainerDied","Data":"94876a0d8909c89fd6dae39ee87f47f3889dd4741e1eb840d81e92b434ee858f"} Aug 04 00:29:50.902452 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.902447 2556 scope.go:117] "RemoveContainer" containerID="ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25" Aug 04 00:29:50.911720 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.911702 2556 scope.go:117] "RemoveContainer" containerID="ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25" Aug 04 00:29:50.911954 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:29:50.911936 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25\": container with ID starting with ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25 not found: ID does not exist" containerID="ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25" Aug 04 00:29:50.912003 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.911965 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25"} err="failed to get container status \"ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25\": rpc error: code = NotFound desc = could not find container \"ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25\": container with ID starting with ac9e30c82e6ff0bf6976ee114e11080f87f9f69fd06872f2ee6495ab5958cb25 not found: ID does not exist" Aug 04 00:29:50.995687 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.995651 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrz8t\" (UniqueName: \"kubernetes.io/projected/c8072820-a124-41d5-8041-3f1ea855cad1-kube-api-access-zrz8t\") pod \"c8072820-a124-41d5-8041-3f1ea855cad1\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " Aug 04 00:29:50.995849 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.995782 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8072820-a124-41d5-8041-3f1ea855cad1-openshift-service-ca\") pod \"c8072820-a124-41d5-8041-3f1ea855cad1\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " Aug 04 00:29:50.995891 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.995857 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls\") pod \"c8072820-a124-41d5-8041-3f1ea855cad1\" (UID: \"c8072820-a124-41d5-8041-3f1ea855cad1\") " Aug 04 00:29:50.996155 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.996132 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8072820-a124-41d5-8041-3f1ea855cad1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8072820-a124-41d5-8041-3f1ea855cad1" (UID: "c8072820-a124-41d5-8041-3f1ea855cad1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:29:50.998155 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.998121 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8072820-a124-41d5-8041-3f1ea855cad1" (UID: "c8072820-a124-41d5-8041-3f1ea855cad1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:29:50.998155 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:50.998137 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8072820-a124-41d5-8041-3f1ea855cad1-kube-api-access-zrz8t" (OuterVolumeSpecName: "kube-api-access-zrz8t") pod "c8072820-a124-41d5-8041-3f1ea855cad1" (UID: "c8072820-a124-41d5-8041-3f1ea855cad1"). InnerVolumeSpecName "kube-api-access-zrz8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:29:51.097056 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:51.097016 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8072820-a124-41d5-8041-3f1ea855cad1-maas-api-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:29:51.097056 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:51.097048 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrz8t\" (UniqueName: \"kubernetes.io/projected/c8072820-a124-41d5-8041-3f1ea855cad1-kube-api-access-zrz8t\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:29:51.097056 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:51.097061 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8072820-a124-41d5-8041-3f1ea855cad1-openshift-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:29:51.909577 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:51.909549 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz" Aug 04 00:29:51.932976 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:51.932945 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz"] Aug 04 00:29:51.937435 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:51.937398 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a51f42-7f954c857b-g2gcz"] Aug 04 00:29:52.421600 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:52.421559 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8072820-a124-41d5-8041-3f1ea855cad1" path="/var/lib/kubelet/pods/c8072820-a124-41d5-8041-3f1ea855cad1/volumes" Aug 04 00:29:52.421994 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:29:52.421973 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e919f839-4c21-412b-a23f-ce80fd2f76ac" path="/var/lib/kubelet/pods/e919f839-4c21-412b-a23f-ce80fd2f76ac/volumes" Aug 04 00:30:00.138108 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.138073 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks"] Aug 04 00:30:00.138586 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.138570 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e919f839-4c21-412b-a23f-ce80fd2f76ac" containerName="revoke-keys" Aug 04 00:30:00.138586 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.138587 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e919f839-4c21-412b-a23f-ce80fd2f76ac" containerName="revoke-keys" Aug 04 00:30:00.138665 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.138605 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8072820-a124-41d5-8041-3f1ea855cad1" containerName="maas-api" Aug 04 00:30:00.138665 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.138614 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8072820-a124-41d5-8041-3f1ea855cad1" containerName="maas-api" Aug 04 00:30:00.138727 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.138707 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8072820-a124-41d5-8041-3f1ea855cad1" containerName="maas-api" Aug 04 00:30:00.138727 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.138717 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e919f839-4c21-412b-a23f-ce80fd2f76ac" containerName="revoke-keys" Aug 04 00:30:00.141691 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.141672 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm"] Aug 04 00:30:00.141839 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.141819 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" Aug 04 00:30:00.144197 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.144176 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:30:00.144560 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.144546 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7"] Aug 04 00:30:00.144667 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.144653 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" Aug 04 00:30:00.147941 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.147625 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966"] Aug 04 00:30:00.148241 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.148223 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7" Aug 04 00:30:00.151492 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.151344 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw"] Aug 04 00:30:00.151763 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.151745 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" Aug 04 00:30:00.155190 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.155150 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks"] Aug 04 00:30:00.155330 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.155260 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw" Aug 04 00:30:00.155699 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.155681 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" Aug 04 00:30:00.156044 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.155692 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7"] Aug 04 00:30:00.158886 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.158866 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966"] Aug 04 00:30:00.168443 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.168420 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" Aug 04 00:30:00.177630 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.177281 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw"] Aug 04 00:30:00.177630 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.177603 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm"] Aug 04 00:30:00.224357 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.224151 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7" Aug 04 00:30:00.233668 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.233462 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" Aug 04 00:30:00.272202 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.272031 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw" Aug 04 00:30:00.371552 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.371451 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks"] Aug 04 00:30:00.382052 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:30:00.381852 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fff719_9832_4ef7_8387_e432e547fe6c.slice/crio-4801b9373b8c01fae8ebf2e9d75fff43e3cc3f7c7489b7716a878d4fc6c8425a WatchSource:0}: Error finding container 4801b9373b8c01fae8ebf2e9d75fff43e3cc3f7c7489b7716a878d4fc6c8425a: Status 404 returned error can't find the container with id 4801b9373b8c01fae8ebf2e9d75fff43e3cc3f7c7489b7716a878d4fc6c8425a Aug 04 00:30:00.413990 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.413964 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm"] Aug 04 00:30:00.418613 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:30:00.417240 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68104ca8_52ac_47d3_a85b_a148f884f690.slice/crio-8e0e6f104cc50587fb0480a92975bffe12d0273afb36676d439e85de23fa6eee WatchSource:0}: Error finding container 8e0e6f104cc50587fb0480a92975bffe12d0273afb36676d439e85de23fa6eee: Status 404 returned error can't find the container with id 8e0e6f104cc50587fb0480a92975bffe12d0273afb36676d439e85de23fa6eee Aug 04 00:30:00.481489 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.481456 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966"] Aug 04 00:30:00.483212 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:30:00.483187 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a9d0ce_7ba3_47eb_8ca5_f1ba90e1e7a3.slice/crio-0b3cd2f044257f65e03b5387f8d3a2851cf6a9f526b1142ed1cd1828d0ba9bac WatchSource:0}: Error finding container 0b3cd2f044257f65e03b5387f8d3a2851cf6a9f526b1142ed1cd1828d0ba9bac: Status 404 returned error can't find the container with id 0b3cd2f044257f65e03b5387f8d3a2851cf6a9f526b1142ed1cd1828d0ba9bac Aug 04 00:30:00.505910 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.505885 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw"] Aug 04 00:30:00.508092 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:30:00.508063 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47aa93ba_4259_480a_b799_1930f3172298.slice/crio-bd62814d351ee3e1457a1f2d238786a8de03020d5ff373200a46fc54ee56eca6 WatchSource:0}: Error finding container bd62814d351ee3e1457a1f2d238786a8de03020d5ff373200a46fc54ee56eca6: Status 404 returned error can't find the container with id bd62814d351ee3e1457a1f2d238786a8de03020d5ff373200a46fc54ee56eca6 Aug 04 00:30:00.668626 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.668602 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7"] Aug 04 00:30:00.669966 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:30:00.669935 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986d8d69_50ac_4e7c_938c_c9bb875d6aa5.slice/crio-eecb138a4e6671331884b64c44d2bd8879a3da9c6ec245665520733a54b5b543 WatchSource:0}: Error finding container eecb138a4e6671331884b64c44d2bd8879a3da9c6ec245665520733a54b5b543: Status 404 returned error can't find the container with id eecb138a4e6671331884b64c44d2bd8879a3da9c6ec245665520733a54b5b543 Aug 04 00:30:00.913570 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.913533 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc"] Aug 04 00:30:00.917216 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.917148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" Aug 04 00:30:00.923034 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.922999 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc"] Aug 04 00:30:00.962342 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.962297 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7" event={"ID":"986d8d69-50ac-4e7c-938c-c9bb875d6aa5","Type":"ContainerStarted","Data":"eecb138a4e6671331884b64c44d2bd8879a3da9c6ec245665520733a54b5b543"} Aug 04 00:30:00.963518 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.963493 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw" event={"ID":"47aa93ba-4259-480a-b799-1930f3172298","Type":"ContainerStarted","Data":"bd62814d351ee3e1457a1f2d238786a8de03020d5ff373200a46fc54ee56eca6"} Aug 04 00:30:00.965009 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.964980 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" event={"ID":"10fff719-9832-4ef7-8387-e432e547fe6c","Type":"ContainerStarted","Data":"def93c6cc4fd2008127c7b34e0e490573518cc3cc82b27144f179f342ca4b4c3"} Aug 04 00:30:00.965112 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.965017 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" event={"ID":"10fff719-9832-4ef7-8387-e432e547fe6c","Type":"ContainerStarted","Data":"4801b9373b8c01fae8ebf2e9d75fff43e3cc3f7c7489b7716a878d4fc6c8425a"} Aug 04 00:30:00.966586 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.966566 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" event={"ID":"f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3","Type":"ContainerStarted","Data":"571aa8489616ee199fb143b9966530b8ef04d5d8dceedbc2c525d101171d0dbb"} Aug 04 00:30:00.966677 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.966606 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" event={"ID":"f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3","Type":"ContainerStarted","Data":"0b3cd2f044257f65e03b5387f8d3a2851cf6a9f526b1142ed1cd1828d0ba9bac"} Aug 04 00:30:00.968135 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.968114 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" event={"ID":"68104ca8-52ac-47d3-a85b-a148f884f690","Type":"ContainerStarted","Data":"1adddefb7aa6e7c080eb57d368b13774bc3da2ba33b687f56440e6e8510a12cc"} Aug 04 00:30:00.968231 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.968149 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" event={"ID":"68104ca8-52ac-47d3-a85b-a148f884f690","Type":"ContainerStarted","Data":"8e0e6f104cc50587fb0480a92975bffe12d0273afb36676d439e85de23fa6eee"} Aug 04 00:30:00.979234 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.979185 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" podStartSLOduration=0.979172319 podStartE2EDuration="979.172319ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:30:00.978654213 +0000 UTC m=+3615.041910646" watchObservedRunningTime="2026-08-04 00:30:00.979172319 +0000 UTC m=+3615.042428749" Aug 04 00:30:00.992391 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:00.992344 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" podStartSLOduration=0.992329202 podStartE2EDuration="992.329202ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:30:00.992164557 +0000 UTC m=+3615.055420987" watchObservedRunningTime="2026-08-04 00:30:00.992329202 +0000 UTC m=+3615.055585665" Aug 04 00:30:01.006838 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.006788 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" podStartSLOduration=1.006773226 podStartE2EDuration="1.006773226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:30:01.004508312 +0000 UTC m=+3615.067764744" watchObservedRunningTime="2026-08-04 00:30:01.006773226 +0000 UTC m=+3615.070029717" Aug 04 00:30:01.093374 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.093329 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203a75de-47f9-4e86-a52a-6287ed74f03b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc\" (UID: \"203a75de-47f9-4e86-a52a-6287ed74f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" Aug 04 00:30:01.195002 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.194908 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203a75de-47f9-4e86-a52a-6287ed74f03b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc\" (UID: \"203a75de-47f9-4e86-a52a-6287ed74f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" Aug 04 00:30:01.195596 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.195576 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203a75de-47f9-4e86-a52a-6287ed74f03b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc\" (UID: \"203a75de-47f9-4e86-a52a-6287ed74f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" Aug 04 00:30:01.239853 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.239815 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" Aug 04 00:30:01.386625 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.386585 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc"] Aug 04 00:30:01.388205 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:30:01.388180 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203a75de_47f9_4e86_a52a_6287ed74f03b.slice/crio-abbb7c1a52732d97da031605c0e7b49244a92a2572ad36372578d1be6bfdf28b WatchSource:0}: Error finding container abbb7c1a52732d97da031605c0e7b49244a92a2572ad36372578d1be6bfdf28b: Status 404 returned error can't find the container with id abbb7c1a52732d97da031605c0e7b49244a92a2572ad36372578d1be6bfdf28b Aug 04 00:30:01.977496 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.977458 2556 generic.go:356] "Generic (PLEG): container finished" podID="47aa93ba-4259-480a-b799-1930f3172298" containerID="15fb076a85ac32252a1c2c9fd1e7df50d8cfdc3f7f88d907f9143b0b371fbed2" exitCode=0 Aug 04 00:30:01.977681 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.977541 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw" event={"ID":"47aa93ba-4259-480a-b799-1930f3172298","Type":"ContainerDied","Data":"15fb076a85ac32252a1c2c9fd1e7df50d8cfdc3f7f88d907f9143b0b371fbed2"} Aug 04 00:30:01.979139 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.979114 2556 generic.go:356] "Generic (PLEG): container finished" podID="10fff719-9832-4ef7-8387-e432e547fe6c" containerID="def93c6cc4fd2008127c7b34e0e490573518cc3cc82b27144f179f342ca4b4c3" exitCode=0 Aug 04 00:30:01.979259 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.979193 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" event={"ID":"10fff719-9832-4ef7-8387-e432e547fe6c","Type":"ContainerDied","Data":"def93c6cc4fd2008127c7b34e0e490573518cc3cc82b27144f179f342ca4b4c3"} Aug 04 00:30:01.980705 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.980685 2556 generic.go:356] "Generic (PLEG): container finished" podID="f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3" containerID="571aa8489616ee199fb143b9966530b8ef04d5d8dceedbc2c525d101171d0dbb" exitCode=0 Aug 04 00:30:01.980813 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.980778 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" event={"ID":"f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3","Type":"ContainerDied","Data":"571aa8489616ee199fb143b9966530b8ef04d5d8dceedbc2c525d101171d0dbb"} Aug 04 00:30:01.982297 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.982279 2556 generic.go:356] "Generic (PLEG): container finished" podID="68104ca8-52ac-47d3-a85b-a148f884f690" containerID="1adddefb7aa6e7c080eb57d368b13774bc3da2ba33b687f56440e6e8510a12cc" exitCode=0 Aug 04 00:30:01.982396 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.982368 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" event={"ID":"68104ca8-52ac-47d3-a85b-a148f884f690","Type":"ContainerDied","Data":"1adddefb7aa6e7c080eb57d368b13774bc3da2ba33b687f56440e6e8510a12cc"} Aug 04 00:30:01.983932 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.983909 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" event={"ID":"203a75de-47f9-4e86-a52a-6287ed74f03b","Type":"ContainerStarted","Data":"7f34fb7ae882b4f706ee8e983eff1972952454dd22674e858f8b03e86346377d"} Aug 04 00:30:01.984013 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.983955 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" event={"ID":"203a75de-47f9-4e86-a52a-6287ed74f03b","Type":"ContainerStarted","Data":"abbb7c1a52732d97da031605c0e7b49244a92a2572ad36372578d1be6bfdf28b"} Aug 04 00:30:01.985336 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.985300 2556 generic.go:356] "Generic (PLEG): container finished" podID="986d8d69-50ac-4e7c-938c-c9bb875d6aa5" containerID="2c7b86d35aa9f517ae4c5307439c7b0eb08197efce5215fc394041db03152785" exitCode=0 Aug 04 00:30:01.985416 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:01.985348 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7" event={"ID":"986d8d69-50ac-4e7c-938c-c9bb875d6aa5","Type":"ContainerDied","Data":"2c7b86d35aa9f517ae4c5307439c7b0eb08197efce5215fc394041db03152785"} Aug 04 00:30:02.031442 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:02.031389 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" podStartSLOduration=2.031372394 podStartE2EDuration="2.031372394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:30:02.030842424 +0000 UTC m=+3616.094098933" watchObservedRunningTime="2026-08-04 00:30:02.031372394 +0000 UTC m=+3616.094628823" Aug 04 00:30:02.993690 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:02.993648 2556 generic.go:356] "Generic (PLEG): container finished" podID="203a75de-47f9-4e86-a52a-6287ed74f03b" containerID="7f34fb7ae882b4f706ee8e983eff1972952454dd22674e858f8b03e86346377d" exitCode=0 Aug 04 00:30:02.993690 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:02.993688 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" event={"ID":"203a75de-47f9-4e86-a52a-6287ed74f03b","Type":"ContainerDied","Data":"7f34fb7ae882b4f706ee8e983eff1972952454dd22674e858f8b03e86346377d"} Aug 04 00:30:03.164811 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:03.164776 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" Aug 04 00:30:03.233415 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:03.233385 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" Aug 04 00:30:03.238858 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:03.238826 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw" Aug 04 00:30:03.301872 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:03.301844 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" Aug 04 00:30:03.304948 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:03.304927 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7" Aug 04 00:30:04.001918 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.001891 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7" Aug 04 00:30:04.002411 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.001912 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7" event={"ID":"986d8d69-50ac-4e7c-938c-c9bb875d6aa5","Type":"ContainerDied","Data":"eecb138a4e6671331884b64c44d2bd8879a3da9c6ec245665520733a54b5b543"} Aug 04 00:30:04.002411 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.001988 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eecb138a4e6671331884b64c44d2bd8879a3da9c6ec245665520733a54b5b543" Aug 04 00:30:04.003244 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.003226 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw" Aug 04 00:30:04.003360 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.003246 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw" event={"ID":"47aa93ba-4259-480a-b799-1930f3172298","Type":"ContainerDied","Data":"bd62814d351ee3e1457a1f2d238786a8de03020d5ff373200a46fc54ee56eca6"} Aug 04 00:30:04.003360 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.003291 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd62814d351ee3e1457a1f2d238786a8de03020d5ff373200a46fc54ee56eca6" Aug 04 00:30:04.004689 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.004669 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" Aug 04 00:30:04.004810 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.004697 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks" event={"ID":"10fff719-9832-4ef7-8387-e432e547fe6c","Type":"ContainerDied","Data":"4801b9373b8c01fae8ebf2e9d75fff43e3cc3f7c7489b7716a878d4fc6c8425a"} Aug 04 00:30:04.004810 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.004741 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4801b9373b8c01fae8ebf2e9d75fff43e3cc3f7c7489b7716a878d4fc6c8425a" Aug 04 00:30:04.006158 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.006142 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" Aug 04 00:30:04.006274 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.006174 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763390-pc966" event={"ID":"f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3","Type":"ContainerDied","Data":"0b3cd2f044257f65e03b5387f8d3a2851cf6a9f526b1142ed1cd1828d0ba9bac"} Aug 04 00:30:04.006274 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.006214 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3cd2f044257f65e03b5387f8d3a2851cf6a9f526b1142ed1cd1828d0ba9bac" Aug 04 00:30:04.007853 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.007833 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" Aug 04 00:30:04.007955 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.007869 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm" event={"ID":"68104ca8-52ac-47d3-a85b-a148f884f690","Type":"ContainerDied","Data":"8e0e6f104cc50587fb0480a92975bffe12d0273afb36676d439e85de23fa6eee"} Aug 04 00:30:04.007955 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.007914 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0e6f104cc50587fb0480a92975bffe12d0273afb36676d439e85de23fa6eee" Aug 04 00:30:04.136487 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.136461 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" Aug 04 00:30:04.329359 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.329257 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203a75de-47f9-4e86-a52a-6287ed74f03b-maas-api-service-ca\") pod \"203a75de-47f9-4e86-a52a-6287ed74f03b\" (UID: \"203a75de-47f9-4e86-a52a-6287ed74f03b\") " Aug 04 00:30:04.329630 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.329605 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203a75de-47f9-4e86-a52a-6287ed74f03b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "203a75de-47f9-4e86-a52a-6287ed74f03b" (UID: "203a75de-47f9-4e86-a52a-6287ed74f03b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:30:04.430912 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:04.430884 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203a75de-47f9-4e86-a52a-6287ed74f03b-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:30:05.015802 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:05.015773 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" Aug 04 00:30:05.016258 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:05.015822 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc" event={"ID":"203a75de-47f9-4e86-a52a-6287ed74f03b","Type":"ContainerDied","Data":"abbb7c1a52732d97da031605c0e7b49244a92a2572ad36372578d1be6bfdf28b"} Aug 04 00:30:05.016258 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:05.015889 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abbb7c1a52732d97da031605c0e7b49244a92a2572ad36372578d1be6bfdf28b" Aug 04 00:30:16.428349 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.428290 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8"] Aug 04 00:30:16.428834 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.428509 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" podUID="488ff5b5-332f-48ad-9962-b70e0760c27e" containerName="maas-api" containerID="cri-o://1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377" gracePeriod=30 Aug 04 00:30:16.456570 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.456538 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw"] Aug 04 00:30:16.458918 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.458897 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8c194a-29763390-tfvvw"] Aug 04 00:30:16.479418 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.479393 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt"] Aug 04 00:30:16.479606 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.479583 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" podUID="b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef" containerName="payload-processing" containerID="cri-o://300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc" gracePeriod=30 Aug 04 00:30:16.486376 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.486348 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h"] Aug 04 00:30:16.486603 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.486570 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" podUID="aab3cb40-4eae-4035-a944-006dd9549a30" containerName="payload-pre-processing" containerID="cri-o://76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863" gracePeriod=30 Aug 04 00:30:16.619032 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.618968 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc"] Aug 04 00:30:16.621816 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.621788 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c194a-fx6vc"] Aug 04 00:30:16.759766 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.759741 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:30:16.777061 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.776547 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:30:16.788014 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.787995 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:30:16.843287 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843252 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f9dw\" (UniqueName: \"kubernetes.io/projected/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-kube-api-access-4f9dw\") pod \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\" (UID: \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\") " Aug 04 00:30:16.843481 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843335 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls\") pod \"488ff5b5-332f-48ad-9962-b70e0760c27e\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " Aug 04 00:30:16.843481 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843445 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab3cb40-4eae-4035-a944-006dd9549a30-plugins-config-volume\") pod \"aab3cb40-4eae-4035-a944-006dd9549a30\" (UID: \"aab3cb40-4eae-4035-a944-006dd9549a30\") " Aug 04 00:30:16.843481 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843474 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-plugins-config-volume\") pod \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\" (UID: \"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef\") " Aug 04 00:30:16.843650 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843495 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/488ff5b5-332f-48ad-9962-b70e0760c27e-openshift-service-ca\") pod \"488ff5b5-332f-48ad-9962-b70e0760c27e\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " Aug 04 00:30:16.843650 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843513 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzq98\" (UniqueName: \"kubernetes.io/projected/488ff5b5-332f-48ad-9962-b70e0760c27e-kube-api-access-zzq98\") pod \"488ff5b5-332f-48ad-9962-b70e0760c27e\" (UID: \"488ff5b5-332f-48ad-9962-b70e0760c27e\") " Aug 04 00:30:16.843650 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843605 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58f8l\" (UniqueName: \"kubernetes.io/projected/aab3cb40-4eae-4035-a944-006dd9549a30-kube-api-access-58f8l\") pod \"aab3cb40-4eae-4035-a944-006dd9549a30\" (UID: \"aab3cb40-4eae-4035-a944-006dd9549a30\") " Aug 04 00:30:16.843881 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843855 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef" (UID: "b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:30:16.843881 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.843863 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488ff5b5-332f-48ad-9962-b70e0760c27e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "488ff5b5-332f-48ad-9962-b70e0760c27e" (UID: "488ff5b5-332f-48ad-9962-b70e0760c27e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:30:16.844218 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.844196 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab3cb40-4eae-4035-a944-006dd9549a30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aab3cb40-4eae-4035-a944-006dd9549a30" (UID: "aab3cb40-4eae-4035-a944-006dd9549a30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:30:16.846886 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.846859 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "488ff5b5-332f-48ad-9962-b70e0760c27e" (UID: "488ff5b5-332f-48ad-9962-b70e0760c27e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:30:16.846886 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.846876 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab3cb40-4eae-4035-a944-006dd9549a30-kube-api-access-58f8l" (OuterVolumeSpecName: "kube-api-access-58f8l") pod "aab3cb40-4eae-4035-a944-006dd9549a30" (UID: "aab3cb40-4eae-4035-a944-006dd9549a30"). InnerVolumeSpecName "kube-api-access-58f8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:30:16.846886 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.846859 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-kube-api-access-4f9dw" (OuterVolumeSpecName: "kube-api-access-4f9dw") pod "b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef" (UID: "b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef"). InnerVolumeSpecName "kube-api-access-4f9dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:30:16.847066 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.846967 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488ff5b5-332f-48ad-9962-b70e0760c27e-kube-api-access-zzq98" (OuterVolumeSpecName: "kube-api-access-zzq98") pod "488ff5b5-332f-48ad-9962-b70e0760c27e" (UID: "488ff5b5-332f-48ad-9962-b70e0760c27e"). InnerVolumeSpecName "kube-api-access-zzq98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:30:16.944607 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.944572 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58f8l\" (UniqueName: \"kubernetes.io/projected/aab3cb40-4eae-4035-a944-006dd9549a30-kube-api-access-58f8l\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:30:16.944607 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.944601 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f9dw\" (UniqueName: \"kubernetes.io/projected/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-kube-api-access-4f9dw\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:30:16.944607 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.944611 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/488ff5b5-332f-48ad-9962-b70e0760c27e-maas-api-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:30:16.944840 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.944619 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab3cb40-4eae-4035-a944-006dd9549a30-plugins-config-volume\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:30:16.944840 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.944628 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef-plugins-config-volume\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:30:16.944840 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.944638 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/488ff5b5-332f-48ad-9962-b70e0760c27e-openshift-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:30:16.944840 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:16.944646 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzq98\" (UniqueName: \"kubernetes.io/projected/488ff5b5-332f-48ad-9962-b70e0760c27e-kube-api-access-zzq98\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:30:17.090647 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.090609 2556 generic.go:356] "Generic (PLEG): container finished" podID="b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef" containerID="300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc" exitCode=0 Aug 04 00:30:17.090845 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.090657 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" Aug 04 00:30:17.090845 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.090699 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" event={"ID":"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef","Type":"ContainerDied","Data":"300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc"} Aug 04 00:30:17.090845 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.090762 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt" event={"ID":"b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef","Type":"ContainerDied","Data":"5c0500532284e35a84f2f48a8e134c61c9d7e33406a4a6912fc2f00e10c78179"} Aug 04 00:30:17.090845 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.090789 2556 scope.go:117] "RemoveContainer" containerID="300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc" Aug 04 00:30:17.092052 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.092028 2556 generic.go:356] "Generic (PLEG): container finished" podID="488ff5b5-332f-48ad-9962-b70e0760c27e" containerID="1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377" exitCode=0 Aug 04 00:30:17.092162 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.092067 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" event={"ID":"488ff5b5-332f-48ad-9962-b70e0760c27e","Type":"ContainerDied","Data":"1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377"} Aug 04 00:30:17.092162 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.092083 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" Aug 04 00:30:17.092162 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.092115 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8" event={"ID":"488ff5b5-332f-48ad-9962-b70e0760c27e","Type":"ContainerDied","Data":"039c17ad581271755749a7823d68ac0baa3849cf923b2101030aad92365f7403"} Aug 04 00:30:17.093540 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.093520 2556 generic.go:356] "Generic (PLEG): container finished" podID="aab3cb40-4eae-4035-a944-006dd9549a30" containerID="76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863" exitCode=0 Aug 04 00:30:17.093638 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.093571 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" Aug 04 00:30:17.093638 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.093588 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" event={"ID":"aab3cb40-4eae-4035-a944-006dd9549a30","Type":"ContainerDied","Data":"76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863"} Aug 04 00:30:17.093728 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.093639 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h" event={"ID":"aab3cb40-4eae-4035-a944-006dd9549a30","Type":"ContainerDied","Data":"22c0c0950584432d6c90cf7522847678381f5422e3674a58f2d2de099bcf1d54"} Aug 04 00:30:17.101723 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.101706 2556 scope.go:117] "RemoveContainer" containerID="300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc" Aug 04 00:30:17.102131 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:30:17.102007 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc\": container with ID starting with 300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc not found: ID does not exist" containerID="300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc" Aug 04 00:30:17.102131 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.102035 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc"} err="failed to get container status \"300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc\": rpc error: code = NotFound desc = could not find container \"300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc\": container with ID starting with 300ccde4e932079c9bdb92bda7a250a904a7e6d097e0a8f92aa72e25f05507bc not found: ID does not exist" Aug 04 00:30:17.102131 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.102051 2556 scope.go:117] "RemoveContainer" containerID="1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377" Aug 04 00:30:17.111270 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.111256 2556 scope.go:117] "RemoveContainer" containerID="1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377" Aug 04 00:30:17.111532 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:30:17.111514 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377\": container with ID starting with 1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377 not found: ID does not exist" containerID="1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377" Aug 04 00:30:17.111581 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.111555 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377"} err="failed to get container status \"1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377\": rpc error: code = NotFound desc = could not find container \"1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377\": container with ID starting with 1256d5ae6e2d3633ea3359f9bd52239f31e9531d8a0e2819a90d12895e03e377 not found: ID does not exist" Aug 04 00:30:17.111581 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.111572 2556 scope.go:117] "RemoveContainer" containerID="76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863" Aug 04 00:30:17.120173 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.120061 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8"] Aug 04 00:30:17.120232 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.120204 2556 scope.go:117] "RemoveContainer" containerID="76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863" Aug 04 00:30:17.120501 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:30:17.120481 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863\": container with ID starting with 76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863 not found: ID does not exist" containerID="76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863" Aug 04 00:30:17.120554 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.120508 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863"} err="failed to get container status \"76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863\": rpc error: code = NotFound desc = could not find container \"76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863\": container with ID starting with 76697e063a7e114940572652a3bb35296bf6a5165e1b0cc45b0fc1e1dea5d863 not found: ID does not exist" Aug 04 00:30:17.124196 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.124177 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8c194a-67cf86f99d-dt5x8"] Aug 04 00:30:17.135391 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.135361 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt"] Aug 04 00:30:17.139920 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.139900 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c194a-6f5bc67f9d-bshnt"] Aug 04 00:30:17.149871 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.149847 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h"] Aug 04 00:30:17.153620 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:17.153598 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c194a-6656c9c856-8985h"] Aug 04 00:30:18.420858 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:18.420821 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203a75de-47f9-4e86-a52a-6287ed74f03b" path="/var/lib/kubelet/pods/203a75de-47f9-4e86-a52a-6287ed74f03b/volumes" Aug 04 00:30:18.421247 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:18.421100 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47aa93ba-4259-480a-b799-1930f3172298" path="/var/lib/kubelet/pods/47aa93ba-4259-480a-b799-1930f3172298/volumes" Aug 04 00:30:18.421457 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:18.421442 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="488ff5b5-332f-48ad-9962-b70e0760c27e" path="/var/lib/kubelet/pods/488ff5b5-332f-48ad-9962-b70e0760c27e/volumes" Aug 04 00:30:18.421768 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:18.421755 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aab3cb40-4eae-4035-a944-006dd9549a30" path="/var/lib/kubelet/pods/aab3cb40-4eae-4035-a944-006dd9549a30/volumes" Aug 04 00:30:18.422377 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:30:18.422362 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef" path="/var/lib/kubelet/pods/b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef/volumes" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.799258 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz"] Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800174 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef" containerName="payload-processing" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800197 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef" containerName="payload-processing" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800221 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10fff719-9832-4ef7-8387-e432e547fe6c" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800230 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="10fff719-9832-4ef7-8387-e432e547fe6c" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800254 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800264 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800281 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47aa93ba-4259-480a-b799-1930f3172298" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800290 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="47aa93ba-4259-480a-b799-1930f3172298" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800300 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203a75de-47f9-4e86-a52a-6287ed74f03b" containerName="revoke-keys" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800328 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="203a75de-47f9-4e86-a52a-6287ed74f03b" containerName="revoke-keys" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800351 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986d8d69-50ac-4e7c-938c-c9bb875d6aa5" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800358 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="986d8d69-50ac-4e7c-938c-c9bb875d6aa5" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800378 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aab3cb40-4eae-4035-a944-006dd9549a30" containerName="payload-pre-processing" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800388 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aab3cb40-4eae-4035-a944-006dd9549a30" containerName="payload-pre-processing" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800396 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68104ca8-52ac-47d3-a85b-a148f884f690" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800403 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="68104ca8-52ac-47d3-a85b-a148f884f690" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800444 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488ff5b5-332f-48ad-9962-b70e0760c27e" containerName="maas-api" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800453 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="488ff5b5-332f-48ad-9962-b70e0760c27e" containerName="maas-api" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800635 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="68104ca8-52ac-47d3-a85b-a148f884f690" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800656 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="47aa93ba-4259-480a-b799-1930f3172298" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800667 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="488ff5b5-332f-48ad-9962-b70e0760c27e" containerName="maas-api" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800682 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cb1a68-b1a4-45ce-9cbe-1fd9f34e11ef" containerName="payload-processing" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800691 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aab3cb40-4eae-4035-a944-006dd9549a30" containerName="payload-pre-processing" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800699 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="986d8d69-50ac-4e7c-938c-c9bb875d6aa5" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800714 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="10fff719-9832-4ef7-8387-e432e547fe6c" containerName="cleanup" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800731 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="203a75de-47f9-4e86-a52a-6287ed74f03b" containerName="revoke-keys" Aug 04 00:31:05.801329 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.800741 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3" containerName="cleanup" Aug 04 00:31:05.804971 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.804945 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" Aug 04 00:31:05.810173 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.810088 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:31:05.811424 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.811399 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz"] Aug 04 00:31:05.877226 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.877189 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d6d6e8-338d-4968-9e24-231b6715fa47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz\" (UID: \"a1d6d6e8-338d-4968-9e24-231b6715fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" Aug 04 00:31:05.978053 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.978014 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d6d6e8-338d-4968-9e24-231b6715fa47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz\" (UID: \"a1d6d6e8-338d-4968-9e24-231b6715fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" Aug 04 00:31:05.978675 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:05.978654 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d6d6e8-338d-4968-9e24-231b6715fa47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz\" (UID: \"a1d6d6e8-338d-4968-9e24-231b6715fa47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" Aug 04 00:31:06.118780 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:06.118684 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" Aug 04 00:31:06.255022 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:06.254998 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz"] Aug 04 00:31:06.256956 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:31:06.256929 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d6d6e8_338d_4968_9e24_231b6715fa47.slice/crio-0bdc4d1b3d122046481db889d5ab478961b35f65f8643ba04573984dd4560024 WatchSource:0}: Error finding container 0bdc4d1b3d122046481db889d5ab478961b35f65f8643ba04573984dd4560024: Status 404 returned error can't find the container with id 0bdc4d1b3d122046481db889d5ab478961b35f65f8643ba04573984dd4560024 Aug 04 00:31:06.379120 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:06.379037 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" event={"ID":"a1d6d6e8-338d-4968-9e24-231b6715fa47","Type":"ContainerStarted","Data":"0bdc4d1b3d122046481db889d5ab478961b35f65f8643ba04573984dd4560024"} Aug 04 00:31:07.385843 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:07.385813 2556 generic.go:356] "Generic (PLEG): container finished" podID="a1d6d6e8-338d-4968-9e24-231b6715fa47" containerID="d25910526bc1dfe5e9187a047e2ecc9c18faac0553cbf795cf1a540b2e5a8ee5" exitCode=0 Aug 04 00:31:07.386217 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:07.385901 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" event={"ID":"a1d6d6e8-338d-4968-9e24-231b6715fa47","Type":"ContainerDied","Data":"d25910526bc1dfe5e9187a047e2ecc9c18faac0553cbf795cf1a540b2e5a8ee5"} Aug 04 00:31:08.524785 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:08.524765 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" Aug 04 00:31:08.596747 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:08.596720 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d6d6e8-338d-4968-9e24-231b6715fa47-maas-api-service-ca\") pod \"a1d6d6e8-338d-4968-9e24-231b6715fa47\" (UID: \"a1d6d6e8-338d-4968-9e24-231b6715fa47\") " Aug 04 00:31:08.597070 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:08.597052 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d6d6e8-338d-4968-9e24-231b6715fa47-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1d6d6e8-338d-4968-9e24-231b6715fa47" (UID: "a1d6d6e8-338d-4968-9e24-231b6715fa47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:31:08.697592 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:08.697520 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d6d6e8-338d-4968-9e24-231b6715fa47-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:09.398821 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:09.398791 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" Aug 04 00:31:09.398991 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:09.398815 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz" event={"ID":"a1d6d6e8-338d-4968-9e24-231b6715fa47","Type":"ContainerDied","Data":"0bdc4d1b3d122046481db889d5ab478961b35f65f8643ba04573984dd4560024"} Aug 04 00:31:09.398991 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:09.398866 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bdc4d1b3d122046481db889d5ab478961b35f65f8643ba04573984dd4560024" Aug 04 00:31:16.291503 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.291463 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl"] Aug 04 00:31:16.292075 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.291729 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" podUID="36d0f13a-0452-48d0-803e-fa728e67e41e" containerName="maas-api" containerID="cri-o://1b3f12b07f114f9336098e75c6ff1d43beaa9ee7a0d7be3947711c57ca81bd88" gracePeriod=30 Aug 04 00:31:16.319885 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.319855 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm"] Aug 04 00:31:16.322143 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.322123 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f96979-29763390-wrrxm"] Aug 04 00:31:16.356011 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.355985 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr"] Aug 04 00:31:16.356200 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.356170 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" podUID="11de2616-b136-432b-a77e-ebf6fee94fb3" containerName="payload-processing" containerID="cri-o://cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be" gracePeriod=30 Aug 04 00:31:16.362458 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.362430 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4"] Aug 04 00:31:16.362681 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.362654 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" podUID="594e8702-5242-4454-aa12-05fd73eb0c4e" containerName="payload-pre-processing" containerID="cri-o://e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea" gracePeriod=30 Aug 04 00:31:16.422536 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.422506 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68104ca8-52ac-47d3-a85b-a148f884f690" path="/var/lib/kubelet/pods/68104ca8-52ac-47d3-a85b-a148f884f690/volumes" Aug 04 00:31:16.442203 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.442179 2556 generic.go:356] "Generic (PLEG): container finished" podID="36d0f13a-0452-48d0-803e-fa728e67e41e" containerID="1b3f12b07f114f9336098e75c6ff1d43beaa9ee7a0d7be3947711c57ca81bd88" exitCode=0 Aug 04 00:31:16.442351 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.442254 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" event={"ID":"36d0f13a-0452-48d0-803e-fa728e67e41e","Type":"ContainerDied","Data":"1b3f12b07f114f9336098e75c6ff1d43beaa9ee7a0d7be3947711c57ca81bd88"} Aug 04 00:31:16.503252 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.502297 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz"] Aug 04 00:31:16.505916 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.505884 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f96979-r2cdz"] Aug 04 00:31:16.618962 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.618932 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:31:16.659494 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.659465 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:31:16.665292 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.665264 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d0f13a-0452-48d0-803e-fa728e67e41e-openshift-service-ca\") pod \"36d0f13a-0452-48d0-803e-fa728e67e41e\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " Aug 04 00:31:16.665465 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.665333 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36d0f13a-0452-48d0-803e-fa728e67e41e-maas-api-tls\") pod \"36d0f13a-0452-48d0-803e-fa728e67e41e\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " Aug 04 00:31:16.665465 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.665377 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cs72\" (UniqueName: \"kubernetes.io/projected/36d0f13a-0452-48d0-803e-fa728e67e41e-kube-api-access-6cs72\") pod \"36d0f13a-0452-48d0-803e-fa728e67e41e\" (UID: \"36d0f13a-0452-48d0-803e-fa728e67e41e\") " Aug 04 00:31:16.665680 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.665650 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d0f13a-0452-48d0-803e-fa728e67e41e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36d0f13a-0452-48d0-803e-fa728e67e41e" (UID: "36d0f13a-0452-48d0-803e-fa728e67e41e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:31:16.665772 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.665740 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d0f13a-0452-48d0-803e-fa728e67e41e-openshift-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:16.668173 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.668148 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36d0f13a-0452-48d0-803e-fa728e67e41e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36d0f13a-0452-48d0-803e-fa728e67e41e" (UID: "36d0f13a-0452-48d0-803e-fa728e67e41e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:31:16.668291 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.668178 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d0f13a-0452-48d0-803e-fa728e67e41e-kube-api-access-6cs72" (OuterVolumeSpecName: "kube-api-access-6cs72") pod "36d0f13a-0452-48d0-803e-fa728e67e41e" (UID: "36d0f13a-0452-48d0-803e-fa728e67e41e"). InnerVolumeSpecName "kube-api-access-6cs72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:31:16.668817 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.668803 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:31:16.766768 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.766742 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gms9\" (UniqueName: \"kubernetes.io/projected/11de2616-b136-432b-a77e-ebf6fee94fb3-kube-api-access-9gms9\") pod \"11de2616-b136-432b-a77e-ebf6fee94fb3\" (UID: \"11de2616-b136-432b-a77e-ebf6fee94fb3\") " Aug 04 00:31:16.766954 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.766802 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594e8702-5242-4454-aa12-05fd73eb0c4e-plugins-config-volume\") pod \"594e8702-5242-4454-aa12-05fd73eb0c4e\" (UID: \"594e8702-5242-4454-aa12-05fd73eb0c4e\") " Aug 04 00:31:16.766954 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.766828 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx7z5\" (UniqueName: \"kubernetes.io/projected/594e8702-5242-4454-aa12-05fd73eb0c4e-kube-api-access-kx7z5\") pod \"594e8702-5242-4454-aa12-05fd73eb0c4e\" (UID: \"594e8702-5242-4454-aa12-05fd73eb0c4e\") " Aug 04 00:31:16.766954 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.766845 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11de2616-b136-432b-a77e-ebf6fee94fb3-plugins-config-volume\") pod \"11de2616-b136-432b-a77e-ebf6fee94fb3\" (UID: \"11de2616-b136-432b-a77e-ebf6fee94fb3\") " Aug 04 00:31:16.767139 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.767014 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36d0f13a-0452-48d0-803e-fa728e67e41e-maas-api-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:16.767139 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.767031 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cs72\" (UniqueName: \"kubernetes.io/projected/36d0f13a-0452-48d0-803e-fa728e67e41e-kube-api-access-6cs72\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:16.767246 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.767225 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594e8702-5242-4454-aa12-05fd73eb0c4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "594e8702-5242-4454-aa12-05fd73eb0c4e" (UID: "594e8702-5242-4454-aa12-05fd73eb0c4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:31:16.767283 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.767239 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11de2616-b136-432b-a77e-ebf6fee94fb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11de2616-b136-432b-a77e-ebf6fee94fb3" (UID: "11de2616-b136-432b-a77e-ebf6fee94fb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:31:16.768917 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.768896 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11de2616-b136-432b-a77e-ebf6fee94fb3-kube-api-access-9gms9" (OuterVolumeSpecName: "kube-api-access-9gms9") pod "11de2616-b136-432b-a77e-ebf6fee94fb3" (UID: "11de2616-b136-432b-a77e-ebf6fee94fb3"). InnerVolumeSpecName "kube-api-access-9gms9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:31:16.769048 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.769023 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594e8702-5242-4454-aa12-05fd73eb0c4e-kube-api-access-kx7z5" (OuterVolumeSpecName: "kube-api-access-kx7z5") pod "594e8702-5242-4454-aa12-05fd73eb0c4e" (UID: "594e8702-5242-4454-aa12-05fd73eb0c4e"). InnerVolumeSpecName "kube-api-access-kx7z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:31:16.867944 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.867863 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594e8702-5242-4454-aa12-05fd73eb0c4e-plugins-config-volume\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:16.867944 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.867892 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx7z5\" (UniqueName: \"kubernetes.io/projected/594e8702-5242-4454-aa12-05fd73eb0c4e-kube-api-access-kx7z5\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:16.867944 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.867905 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11de2616-b136-432b-a77e-ebf6fee94fb3-plugins-config-volume\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:16.867944 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:16.867917 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gms9\" (UniqueName: \"kubernetes.io/projected/11de2616-b136-432b-a77e-ebf6fee94fb3-kube-api-access-9gms9\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:17.448389 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.448357 2556 generic.go:356] "Generic (PLEG): container finished" podID="594e8702-5242-4454-aa12-05fd73eb0c4e" containerID="e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea" exitCode=0 Aug 04 00:31:17.448822 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.448401 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" Aug 04 00:31:17.448822 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.448440 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" event={"ID":"594e8702-5242-4454-aa12-05fd73eb0c4e","Type":"ContainerDied","Data":"e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea"} Aug 04 00:31:17.448822 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.448491 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4" event={"ID":"594e8702-5242-4454-aa12-05fd73eb0c4e","Type":"ContainerDied","Data":"a4de6d5c21558c784df2761f9d9da0581214e8953a91fcf6d6218225b4019529"} Aug 04 00:31:17.448822 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.448519 2556 scope.go:117] "RemoveContainer" containerID="e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea" Aug 04 00:31:17.449917 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.449824 2556 generic.go:356] "Generic (PLEG): container finished" podID="11de2616-b136-432b-a77e-ebf6fee94fb3" containerID="cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be" exitCode=0 Aug 04 00:31:17.449917 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.449865 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" Aug 04 00:31:17.450052 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.449915 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" event={"ID":"11de2616-b136-432b-a77e-ebf6fee94fb3","Type":"ContainerDied","Data":"cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be"} Aug 04 00:31:17.450052 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.449965 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr" event={"ID":"11de2616-b136-432b-a77e-ebf6fee94fb3","Type":"ContainerDied","Data":"b21bd2e50cbed92e8a107376536549fc103dfb781afd8610e4235e926b9ae599"} Aug 04 00:31:17.451545 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.451515 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" event={"ID":"36d0f13a-0452-48d0-803e-fa728e67e41e","Type":"ContainerDied","Data":"19ad29e48cae7f0dcaa292d73f6934e7f0a5422aeb72b24d1ba1a26282b33bdc"} Aug 04 00:31:17.451647 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.451527 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl" Aug 04 00:31:17.459929 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.459916 2556 scope.go:117] "RemoveContainer" containerID="e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea" Aug 04 00:31:17.460217 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:31:17.460194 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea\": container with ID starting with e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea not found: ID does not exist" containerID="e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea" Aug 04 00:31:17.460285 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.460226 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea"} err="failed to get container status \"e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea\": rpc error: code = NotFound desc = could not find container \"e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea\": container with ID starting with e6027c557d1f64110124582f32bf1a3e210c47a5568cc2e0f10221df7d265eea not found: ID does not exist" Aug 04 00:31:17.460285 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.460243 2556 scope.go:117] "RemoveContainer" containerID="cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be" Aug 04 00:31:17.468749 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.468734 2556 scope.go:117] "RemoveContainer" containerID="cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be" Aug 04 00:31:17.468990 ip-10-0-136-126 kubenswrapper[2556]: E0804 00:31:17.468974 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be\": container with ID starting with cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be not found: ID does not exist" containerID="cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be" Aug 04 00:31:17.469042 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.468994 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be"} err="failed to get container status \"cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be\": rpc error: code = NotFound desc = could not find container \"cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be\": container with ID starting with cf5702f1fa4350f6916f909f6b17478d68f7236292fb268daa629bd48c7505be not found: ID does not exist" Aug 04 00:31:17.469042 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.469010 2556 scope.go:117] "RemoveContainer" containerID="1b3f12b07f114f9336098e75c6ff1d43beaa9ee7a0d7be3947711c57ca81bd88" Aug 04 00:31:17.476772 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.476750 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4"] Aug 04 00:31:17.480930 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.480912 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f96979-f78d57cb-r2wc4"] Aug 04 00:31:17.491365 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.491344 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr"] Aug 04 00:31:17.493443 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.493426 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f96979-7648d66d6d-cf2sr"] Aug 04 00:31:17.502631 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.502612 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl"] Aug 04 00:31:17.508980 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:17.508959 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f96979-64f7869cf7-tmfwl"] Aug 04 00:31:18.421431 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:18.421391 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11de2616-b136-432b-a77e-ebf6fee94fb3" path="/var/lib/kubelet/pods/11de2616-b136-432b-a77e-ebf6fee94fb3/volumes" Aug 04 00:31:18.421802 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:18.421789 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36d0f13a-0452-48d0-803e-fa728e67e41e" path="/var/lib/kubelet/pods/36d0f13a-0452-48d0-803e-fa728e67e41e/volumes" Aug 04 00:31:18.422097 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:18.422085 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="594e8702-5242-4454-aa12-05fd73eb0c4e" path="/var/lib/kubelet/pods/594e8702-5242-4454-aa12-05fd73eb0c4e/volumes" Aug 04 00:31:18.422386 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:18.422374 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d6d6e8-338d-4968-9e24-231b6715fa47" path="/var/lib/kubelet/pods/a1d6d6e8-338d-4968-9e24-231b6715fa47/volumes" Aug 04 00:31:23.606576 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606538 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf"] Aug 04 00:31:23.606945 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606915 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11de2616-b136-432b-a77e-ebf6fee94fb3" containerName="payload-processing" Aug 04 00:31:23.606945 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606926 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11de2616-b136-432b-a77e-ebf6fee94fb3" containerName="payload-processing" Aug 04 00:31:23.606945 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606935 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d0f13a-0452-48d0-803e-fa728e67e41e" containerName="maas-api" Aug 04 00:31:23.606945 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606941 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d0f13a-0452-48d0-803e-fa728e67e41e" containerName="maas-api" Aug 04 00:31:23.607076 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606948 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594e8702-5242-4454-aa12-05fd73eb0c4e" containerName="payload-pre-processing" Aug 04 00:31:23.607076 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606956 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="594e8702-5242-4454-aa12-05fd73eb0c4e" containerName="payload-pre-processing" Aug 04 00:31:23.607076 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606971 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d6d6e8-338d-4968-9e24-231b6715fa47" containerName="revoke-keys" Aug 04 00:31:23.607076 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.606976 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d6d6e8-338d-4968-9e24-231b6715fa47" containerName="revoke-keys" Aug 04 00:31:23.607076 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.607027 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="594e8702-5242-4454-aa12-05fd73eb0c4e" containerName="payload-pre-processing" Aug 04 00:31:23.607076 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.607034 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11de2616-b136-432b-a77e-ebf6fee94fb3" containerName="payload-processing" Aug 04 00:31:23.607076 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.607042 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d0f13a-0452-48d0-803e-fa728e67e41e" containerName="maas-api" Aug 04 00:31:23.607076 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.607049 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d6d6e8-338d-4968-9e24-231b6715fa47" containerName="revoke-keys" Aug 04 00:31:23.611677 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.611658 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" Aug 04 00:31:23.614486 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.614456 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x6w2\"" Aug 04 00:31:23.616672 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.616646 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf"] Aug 04 00:31:23.725988 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.725948 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f61085-1804-4125-ad68-52e29228c81a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf\" (UID: \"c5f61085-1804-4125-ad68-52e29228c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" Aug 04 00:31:23.826600 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.826560 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f61085-1804-4125-ad68-52e29228c81a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf\" (UID: \"c5f61085-1804-4125-ad68-52e29228c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" Aug 04 00:31:23.827158 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.827139 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f61085-1804-4125-ad68-52e29228c81a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf\" (UID: \"c5f61085-1804-4125-ad68-52e29228c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" Aug 04 00:31:23.936401 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:23.936324 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" Aug 04 00:31:24.071167 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:24.071137 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf"] Aug 04 00:31:24.072258 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:31:24.072231 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f61085_1804_4125_ad68_52e29228c81a.slice/crio-fe1237fde20b17b1460e105d555cd3b6603386a62429564e0f422dd75b19908c WatchSource:0}: Error finding container fe1237fde20b17b1460e105d555cd3b6603386a62429564e0f422dd75b19908c: Status 404 returned error can't find the container with id fe1237fde20b17b1460e105d555cd3b6603386a62429564e0f422dd75b19908c Aug 04 00:31:24.500737 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:24.500645 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" event={"ID":"c5f61085-1804-4125-ad68-52e29228c81a","Type":"ContainerStarted","Data":"aa7a7acc896e6802b05935fc82bb932ff7a718518a6b9844718a3d1bf959b12b"} Aug 04 00:31:24.500737 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:24.500697 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" event={"ID":"c5f61085-1804-4125-ad68-52e29228c81a","Type":"ContainerStarted","Data":"fe1237fde20b17b1460e105d555cd3b6603386a62429564e0f422dd75b19908c"} Aug 04 00:31:24.518720 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:24.518674 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" podStartSLOduration=1.5186597590000002 podStartE2EDuration="1.518659759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:31:24.516420075 +0000 UTC m=+3698.579676506" watchObservedRunningTime="2026-08-04 00:31:24.518659759 +0000 UTC m=+3698.581916189" Aug 04 00:31:25.507418 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:25.507378 2556 generic.go:356] "Generic (PLEG): container finished" podID="c5f61085-1804-4125-ad68-52e29228c81a" containerID="aa7a7acc896e6802b05935fc82bb932ff7a718518a6b9844718a3d1bf959b12b" exitCode=0 Aug 04 00:31:25.507893 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:25.507467 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" event={"ID":"c5f61085-1804-4125-ad68-52e29228c81a","Type":"ContainerDied","Data":"aa7a7acc896e6802b05935fc82bb932ff7a718518a6b9844718a3d1bf959b12b"} Aug 04 00:31:26.649554 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:26.649527 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" Aug 04 00:31:26.755381 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:26.755346 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f61085-1804-4125-ad68-52e29228c81a-maas-api-service-ca\") pod \"c5f61085-1804-4125-ad68-52e29228c81a\" (UID: \"c5f61085-1804-4125-ad68-52e29228c81a\") " Aug 04 00:31:26.755745 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:26.755724 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f61085-1804-4125-ad68-52e29228c81a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5f61085-1804-4125-ad68-52e29228c81a" (UID: "c5f61085-1804-4125-ad68-52e29228c81a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:31:26.856376 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:26.856263 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f61085-1804-4125-ad68-52e29228c81a-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Aug 04 00:31:27.520412 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:27.520381 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" Aug 04 00:31:27.520591 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:27.520415 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf" event={"ID":"c5f61085-1804-4125-ad68-52e29228c81a","Type":"ContainerDied","Data":"fe1237fde20b17b1460e105d555cd3b6603386a62429564e0f422dd75b19908c"} Aug 04 00:31:27.520591 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:27.520459 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe1237fde20b17b1460e105d555cd3b6603386a62429564e0f422dd75b19908c" Aug 04 00:31:34.041401 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:34.041368 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks"] Aug 04 00:31:34.043578 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:34.043556 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1769a3-29763390-bsmks"] Aug 04 00:31:34.209902 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:34.209871 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf"] Aug 04 00:31:34.214984 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:34.214951 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1769a3-ffdjf"] Aug 04 00:31:34.421848 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:34.421771 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10fff719-9832-4ef7-8387-e432e547fe6c" path="/var/lib/kubelet/pods/10fff719-9832-4ef7-8387-e432e547fe6c/volumes" Aug 04 00:31:34.422091 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:34.422078 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f61085-1804-4125-ad68-52e29228c81a" path="/var/lib/kubelet/pods/c5f61085-1804-4125-ad68-52e29228c81a/volumes" Aug 04 00:31:51.754048 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:51.754011 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7"] Aug 04 00:31:51.756207 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:51.756183 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-98f978-29763390-d2lh7"] Aug 04 00:31:52.420724 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:31:52.420681 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="986d8d69-50ac-4e7c-938c-c9bb875d6aa5" path="/var/lib/kubelet/pods/986d8d69-50ac-4e7c-938c-c9bb875d6aa5/volumes" Aug 04 00:32:04.829869 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:04.829838 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-t5t2j_3b2f3444-26a7-41ca-986b-e808796cf632/manager/0.log" Aug 04 00:32:05.183689 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:05.183617 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6869c6d9bb-mk77v_a8a09e21-d143-472d-bfc4-58719aef68f2/copy-manifests/0.log" Aug 04 00:32:05.189206 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:05.189182 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6869c6d9bb-mk77v_a8a09e21-d143-472d-bfc4-58719aef68f2/manager/0.log" Aug 04 00:32:05.716529 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:05.716505 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29763375-blkwp_cec2d0dd-720d-41a8-ae1d-e6c8936f6f97/cleanup/0.log" Aug 04 00:32:05.824925 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:05.824894 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29763390-pc966_f7a9d0ce-7ba3-47eb-8ca5-f1ba90e1e7a3/cleanup/0.log" Aug 04 00:32:05.940448 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:05.940418 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-q4pv4_893acca7-29d3-4d1f-9753-eaad035d1b45/postgres/0.log" Aug 04 00:32:07.356774 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:07.356737 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-b9m5v_85cee131-c0ac-464d-8b74-8957f00bee1b/manager/0.log" Aug 04 00:32:07.799952 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:07.799916 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qbbfm_d4546db1-5f37-4b78-a0e9-37f65ece8d40/discovery/0.log" Aug 04 00:32:08.830565 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:08.830541 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gqjnc_c880f829-6827-4943-bc05-2c53d40d1fd7/main/0.log" Aug 04 00:32:09.159767 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:09.159686 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwvbbw_8a071b68-6494-4985-bd81-1974b2cd9cb7/main/0.log" Aug 04 00:32:18.043438 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:18.043403 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nd4xg_36bfdb11-2080-4407-9208-2d5ef6995e91/global-pull-secret-syncer/0.log" Aug 04 00:32:18.168270 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:18.168235 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4zqgb_d8903faa-2d47-47a2-a7c0-ef8d5f7fb148/konnectivity-agent/0.log" Aug 04 00:32:18.220675 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:18.220641 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-126.ec2.internal_d8b7e0082a8d432502c091a2ad7a9c78/haproxy/0.log" Aug 04 00:32:23.684552 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:23.684520 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-b9m5v_85cee131-c0ac-464d-8b74-8957f00bee1b/manager/0.log" Aug 04 00:32:25.361586 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.361553 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9krsz_8408b47a-d8ca-4b65-99f8-344a836926cb/kube-state-metrics/0.log" Aug 04 00:32:25.382798 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.382770 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9krsz_8408b47a-d8ca-4b65-99f8-344a836926cb/kube-rbac-proxy-main/0.log" Aug 04 00:32:25.404391 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.404371 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9krsz_8408b47a-d8ca-4b65-99f8-344a836926cb/kube-rbac-proxy-self/0.log" Aug 04 00:32:25.464282 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.464253 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-vgn9j_86336890-d32f-4d81-9b89-dcd8adeacb93/monitoring-plugin/0.log" Aug 04 00:32:25.614458 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.614398 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlcrf_5d1b4b01-e938-4e2d-97b4-541bebc29b50/node-exporter/0.log" Aug 04 00:32:25.638821 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.638800 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlcrf_5d1b4b01-e938-4e2d-97b4-541bebc29b50/kube-rbac-proxy/0.log" Aug 04 00:32:25.662422 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.662401 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlcrf_5d1b4b01-e938-4e2d-97b4-541bebc29b50/init-textfile/0.log" Aug 04 00:32:25.788649 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.788621 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-bgccp_ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6/kube-rbac-proxy-main/0.log" Aug 04 00:32:25.811742 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.811718 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-bgccp_ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6/kube-rbac-proxy-self/0.log" Aug 04 00:32:25.832619 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:25.832597 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-bgccp_ddd1d13e-ebf3-4f1a-a336-8d5bb734f8c6/openshift-state-metrics/0.log" Aug 04 00:32:26.208064 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:26.208036 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f667797f-zcbc9_6dec7799-779d-4f15-9763-311dbb23ff21/thanos-query/0.log" Aug 04 00:32:26.229776 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:26.229751 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f667797f-zcbc9_6dec7799-779d-4f15-9763-311dbb23ff21/kube-rbac-proxy-web/0.log" Aug 04 00:32:26.250643 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:26.250612 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f667797f-zcbc9_6dec7799-779d-4f15-9763-311dbb23ff21/kube-rbac-proxy/0.log" Aug 04 00:32:26.271967 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:26.271941 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f667797f-zcbc9_6dec7799-779d-4f15-9763-311dbb23ff21/prom-label-proxy/0.log" Aug 04 00:32:26.293576 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:26.293551 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f667797f-zcbc9_6dec7799-779d-4f15-9763-311dbb23ff21/kube-rbac-proxy-rules/0.log" Aug 04 00:32:26.320496 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:26.320473 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9f667797f-zcbc9_6dec7799-779d-4f15-9763-311dbb23ff21/kube-rbac-proxy-metrics/0.log" Aug 04 00:32:27.558161 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:27.558136 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-9cxjw_d7aec628-af56-43fe-968c-81a79d5517f1/networking-console-plugin/0.log" Aug 04 00:32:28.149951 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.149923 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/1.log" Aug 04 00:32:28.154707 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.154690 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-zp9bh_4100d470-402e-4f60-aecc-06e52fc7140b/console-operator/2.log" Aug 04 00:32:28.344573 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.344539 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7"] Aug 04 00:32:28.344948 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.344935 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f61085-1804-4125-ad68-52e29228c81a" containerName="revoke-keys" Aug 04 00:32:28.344991 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.344950 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f61085-1804-4125-ad68-52e29228c81a" containerName="revoke-keys" Aug 04 00:32:28.345021 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.345016 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f61085-1804-4125-ad68-52e29228c81a" containerName="revoke-keys" Aug 04 00:32:28.348089 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.348076 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.351057 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.351033 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b95cg\"/\"kube-root-ca.crt\"" Aug 04 00:32:28.351545 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.351525 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b95cg\"/\"openshift-service-ca.crt\"" Aug 04 00:32:28.351747 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.351723 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b95cg\"/\"default-dockercfg-zqc6c\"" Aug 04 00:32:28.363279 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.363250 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7"] Aug 04 00:32:28.469546 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.469522 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4t5\" (UniqueName: \"kubernetes.io/projected/70d73762-a176-4bf7-a35a-5bf0573e10a6-kube-api-access-gd4t5\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.469708 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.469561 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-proc\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.469708 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.469636 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-sys\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.469708 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.469681 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-lib-modules\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.469708 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.469703 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-podres\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.570915 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.570884 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-sys\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.571491 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.570936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-lib-modules\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.571491 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.570970 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-podres\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.571491 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.571017 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4t5\" (UniqueName: \"kubernetes.io/projected/70d73762-a176-4bf7-a35a-5bf0573e10a6-kube-api-access-gd4t5\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.571491 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.571018 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-sys\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.571491 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.571050 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-proc\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.571491 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.571093 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-lib-modules\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.571491 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.571159 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-proc\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.571491 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.571168 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/70d73762-a176-4bf7-a35a-5bf0573e10a6-podres\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.578872 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.578851 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4t5\" (UniqueName: \"kubernetes.io/projected/70d73762-a176-4bf7-a35a-5bf0573e10a6-kube-api-access-gd4t5\") pod \"perf-node-gather-daemonset-8cjk7\" (UID: \"70d73762-a176-4bf7-a35a-5bf0573e10a6\") " pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.664468 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.664440 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:28.800555 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.800526 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7"] Aug 04 00:32:28.802661 ip-10-0-136-126 kubenswrapper[2556]: W0804 00:32:28.802633 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod70d73762_a176_4bf7_a35a_5bf0573e10a6.slice/crio-f94b3d28d678ce641ebe408e5d86c86fd09650c50ac5ccf8dad1715fa17721f7 WatchSource:0}: Error finding container f94b3d28d678ce641ebe408e5d86c86fd09650c50ac5ccf8dad1715fa17721f7: Status 404 returned error can't find the container with id f94b3d28d678ce641ebe408e5d86c86fd09650c50ac5ccf8dad1715fa17721f7 Aug 04 00:32:28.832409 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.832389 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bdbb769bb-9vfpm_031ccbfe-ab72-4869-82f1-076f05d4dcf4/console/0.log" Aug 04 00:32:28.887345 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:28.887292 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" event={"ID":"70d73762-a176-4bf7-a35a-5bf0573e10a6","Type":"ContainerStarted","Data":"f94b3d28d678ce641ebe408e5d86c86fd09650c50ac5ccf8dad1715fa17721f7"} Aug 04 00:32:29.526161 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:29.526129 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-bgstd_c39dd3f1-1988-4c1e-81ba-c57cfb0a75f5/volume-data-source-validator/0.log" Aug 04 00:32:29.894245 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:29.894140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" event={"ID":"70d73762-a176-4bf7-a35a-5bf0573e10a6","Type":"ContainerStarted","Data":"5954827637a10e1d7c24a51f6f183922233f81018544491738bbf6e73b8f2c49"} Aug 04 00:32:29.894245 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:29.894214 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:29.909723 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:29.909676 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" podStartSLOduration=1.909662413 podStartE2EDuration="1.909662413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:32:29.908987371 +0000 UTC m=+3763.972243802" watchObservedRunningTime="2026-08-04 00:32:29.909662413 +0000 UTC m=+3763.972918842" Aug 04 00:32:30.684216 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:30.684186 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w4tdk_2062046e-c1ef-42fb-b341-fab17c10c653/dns/0.log" Aug 04 00:32:30.704653 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:30.704625 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w4tdk_2062046e-c1ef-42fb-b341-fab17c10c653/kube-rbac-proxy/0.log" Aug 04 00:32:30.728588 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:30.728560 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7r62k_347d49d7-131d-4942-82ea-6ce44f73495d/dns-node-resolver/0.log" Aug 04 00:32:31.381082 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:31.381055 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nsl2l_30f44684-cb92-47de-9a0f-ffc6ca6346e6/node-ca/0.log" Aug 04 00:32:32.566938 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:32.566902 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qbbfm_d4546db1-5f37-4b78-a0e9-37f65ece8d40/discovery/0.log" Aug 04 00:32:33.526277 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:33.526252 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9wh8d_5610d18a-21f3-4cab-be04-b7a8119b6105/serve-healthcheck-canary/0.log" Aug 04 00:32:34.072345 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:34.072291 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qhrx_95547717-0f6c-42dc-9cc2-219289a032f7/kube-rbac-proxy/0.log" Aug 04 00:32:34.094030 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:34.094000 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qhrx_95547717-0f6c-42dc-9cc2-219289a032f7/exporter/0.log" Aug 04 00:32:34.114930 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:34.114902 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5qhrx_95547717-0f6c-42dc-9cc2-219289a032f7/extractor/0.log" Aug 04 00:32:35.911548 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:35.911518 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b95cg/perf-node-gather-daemonset-8cjk7" Aug 04 00:32:36.853902 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:36.853805 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-t5t2j_3b2f3444-26a7-41ca-986b-e808796cf632/manager/0.log" Aug 04 00:32:37.089185 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:37.089162 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6869c6d9bb-mk77v_a8a09e21-d143-472d-bfc4-58719aef68f2/manager/0.log" Aug 04 00:32:37.110105 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:37.110044 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6869c6d9bb-mk77v_a8a09e21-d143-472d-bfc4-58719aef68f2/copy-manifests/0.log" Aug 04 00:32:38.588279 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:38.588253 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d64bf5b4d-jv4dc_6e1292b8-1180-4bb0-836d-0ad18b629d4a/manager/0.log" Aug 04 00:32:39.410375 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:39.410330 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qbbfm_d4546db1-5f37-4b78-a0e9-37f65ece8d40/discovery/0.log" Aug 04 00:32:41.097528 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:41.097497 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qbbfm_d4546db1-5f37-4b78-a0e9-37f65ece8d40/discovery/0.log" Aug 04 00:32:43.505899 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:43.505860 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-kghsl_5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe/migrator/0.log" Aug 04 00:32:43.525591 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:43.525562 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-kghsl_5b4d21d3-12ef-4c7d-a4d0-72f5869b3dfe/graceful-termination/0.log" Aug 04 00:32:44.844259 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:44.844219 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7tb9z_95fc751e-3c56-4f22-97c9-e0664b53b97f/kube-multus/0.log" Aug 04 00:32:44.867539 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:44.867515 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxfpf_73bcb60f-5b25-4b51-8d80-4bc2f5276f12/kube-multus-additional-cni-plugins/0.log" Aug 04 00:32:44.888455 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:44.888437 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxfpf_73bcb60f-5b25-4b51-8d80-4bc2f5276f12/egress-router-binary-copy/0.log" Aug 04 00:32:44.908416 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:44.908397 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxfpf_73bcb60f-5b25-4b51-8d80-4bc2f5276f12/cni-plugins/0.log" Aug 04 00:32:44.928078 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:44.928056 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxfpf_73bcb60f-5b25-4b51-8d80-4bc2f5276f12/bond-cni-plugin/0.log" Aug 04 00:32:44.947521 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:44.947502 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxfpf_73bcb60f-5b25-4b51-8d80-4bc2f5276f12/routeoverride-cni/0.log" Aug 04 00:32:44.967415 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:44.967393 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxfpf_73bcb60f-5b25-4b51-8d80-4bc2f5276f12/whereabouts-cni-bincopy/0.log" Aug 04 00:32:44.987570 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:44.987550 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxfpf_73bcb60f-5b25-4b51-8d80-4bc2f5276f12/whereabouts-cni/0.log" Aug 04 00:32:45.421698 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:45.421673 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jsxr8_29374835-d52e-403b-9834-071b2f9bda81/network-metrics-daemon/0.log" Aug 04 00:32:45.440178 ip-10-0-136-126 kubenswrapper[2556]: I0804 00:32:45.440154 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jsxr8_29374835-d52e-403b-9834-071b2f9bda81/kube-rbac-proxy/0.log"